HTML Entities Problem

Submitted by Bangkok Bob on Mon, 2007-11-26 10:27 in Magic Parser

Hi David,

I am trying display the feed at http://www.dcothai.com/rss-news.php (encoding=ISO-8859-1) for display on a webpage (charset=ISO-8859-1) but would like some ideas on how to clean up the weird html entities/strange characters in the title and description fields.

Best regards,
Bangkok Bob

Hi Bob, I've just had a look

Submitted by support on Mon, 2007-11-26 11:38

Hi Bob,

I've just had a look at the feed. I wrote the following test script which simply displays the content "as-is", and the entities are all displayed correctly (at least in my browser):

http://www.magicparser.com/examples/dcothai.php

Here is the source code:

<?php
  header("Content-Type: text/html;charset=iso-8859-1");
  require("MagicParser.php");
  function myRecordHandler($item)
  {
    print "<h2>".$item["TITLE"]."</h2>";
    print "<p>".$item["DESCRIPTION"]."</p>";
  }  
  MagicParser_parse("dcothai.xml","myRecordHandler","xml|RSS/CHANNEL/ITEM/");
?>

(I saved the response from your URL as dcothai.xml in the same directory as the script)

If you're not seeing the same results, make sure that the correct character set header is being sent by your script (see the header() line in the above code)...

Cheers,
David.

Hi David, Thanks for your

Submitted by Bangkok Bob on Mon, 2007-11-26 22:48

Hi David,

Thanks for your prompt response, as always.

Maybe it's a browser issue, then. I go to http://www.dcothai.com/rss-news.php then view source in FireFox so I can save (for subsequent upload to PT) but the ampersand is being altered thus causing problems later. I tried IE but that won't show XML only a blank page and no view source.

<item>

<title>Les m&amp;#233;moires d&amp;#39;un d&amp;#233;tective priv&amp;#233; &amp;#224; Bang</title>

<link>http://www.dcothai.com/product_info.php?products_id=816</link>

<guid>http://www.dcothai.com/product_info.php?products_id=816</guid>

<description>&lt;P class=producthead&gt;Les m&amp;#233;moires d&amp;#39;un d&amp;#233;tective priv&amp;#233; &amp;#224; Bangkok&lt;/p&gt;&lt;P class=productheadtwo&gt;Histoires vraies tir&amp;#233;es des dossiers de Warren Olson&lt;/p&gt;&lt;p&gt;Quand vous n&amp;#39&amp;#234;tes pas l&amp;#224;, que fait votre copine ? www. Thaiprivateeye .com&lt;/p&gt;&lt;p&gt;Les filles de bar avec un double emploi, les &amp;#233;pouses suspicieuses, l&amp;#39amante lesbienne, voil&amp;#224; ce qu&amp;#39&amp;#233;taient les journ&amp;#233;es du d&amp;#233;tective priv&amp;#233; de Bangkok Warren Olson.&lt;/p&gt;&lt;p&gt;Pendant plus de dix ans Olson a parcouru les petites rues de la &amp;#39Grosse Mangue&amp;#39. Parlant parfaitement le Tha&amp;#239; et le Khmer, il &amp;#233;tait capable d&amp;#39aller l&amp;#224; o&amp;#249; les autres d&amp;#233;tectives n&amp;#39osaient pas tra&amp;#238;ner.&lt;/p&gt;&lt;p&gt;Ces clients &amp;#233;taient des occidentaux qui avaient perdu leur coeur -- et leurs &amp;#233;conomies -- avec des filles de bar affam&amp;#233;es. Mais il eut aussi pas mal de clients tha&amp;#239;s dont une charmante vieille dame qui avait &amp;#233;t&amp;#233; d&amp;#233;trouss&amp;#233;e par un arnaqueur chr&amp;#233;tien et aussi une fille tha&amp;#239;e &amp;#224; laquelle un ancien amant faisait du chantage.&lt;/p&gt;&lt;p&gt;Personne ne sait mieux que Warren Olson les combines que les filles de bar peuvent employer pour que les clients occidentaux se d&amp;#233;partissent de leur argent durement gagn&amp;#233;. Maintenant qu&amp;#39il a arr&amp;#234;t&amp;#233;, il a envie de partager ses exp&amp;#233;riences avec le reste du monde. Ses grands moments de sagesse sont :&lt;br&gt;- D&amp;#233;finition d&amp;#39une fille de bar : &lt;br&gt;rus&amp;#233;e comme un renard mais avec la cervelle d&amp;#39un poisson rouge. &lt;br&gt;- Quand vous avez comme copine une fille de bar, vous ne la perdez jamais. Vous perdez seulement quelquefois votre place dans la queue.&lt;/p&gt;&lt;p&gt;Les histoires sont bas&amp;#233;es sur les dossiers de Warren Olson. Pour prot&amp;#233;ger les innocents et les coupables, elles sont arrang&amp;#233;es par l&amp;#39auteur de best-sellers Stephen Leather. Olson est maintenant reparti dans son pays natal, la Nouvelle Z&amp;#233;lande, avec sa femme tha&amp;#239;e et leur fille mais l&amp;#39agence qu&amp;#39il a cr&amp;#233;e est toujours ouverte. On peut avoir des infos sur : www. thaiprivateeye .com&lt;/p&gt;&lt;p&gt;Paperback&lt;br&gt;Bangkok 2006&lt;br&gt;Bamboo Sinfonia&lt;br&gt;ISBN 9789748284033&lt;br&gt;305 pages&lt;/p&gt;</description>

<ecommerce:listPrice>$16.45</ecommerce:listPrice>

<ecommerce:SKU>20004186</ecommerce:SKU>

<category>Books Thailand</category>

</item>

When I look at the example you parsed at http://www.magicparser.com/examples/dcothai.php, FireFox shows a validation error.

Warning: entity "&#39" doesn't end in ';'

Best regards,
Bangkok Bob

Hi Bob, The &#39 entiti

Submitted by support on Tue, 2007-11-27 09:52

Hi Bob,

The &#39 entiti seems to be a strange one as although it is not encoded correctly in the feed, FireFox displays it correctly as the ' character.

The best way to deal with this if you want to fix it up, I think is to do a str_replace on this string when reading the feed, for example:

$record["DESCRIPTION"] = str_replace("&#39","'",$record["DESCRIPTION"]);

To save the feed for loading into PT, if you view with a web browser and then go View > Source you can normally then save to disk using File > Save As...

If the problem you are having is later on and you are importing the feed using Price Tapestry, the entities will be broken by default because the "&", "#" and ";" characters are all stripped from fields during import. This behavior is easily changed - see the following thread for the code changes...

http://www.pricetapestry.com/node/1524

Cheers,
David.

Hi David, Thanks for your

Submitted by Bangkok Bob on Thu, 2007-11-29 01:40

Hi David,

Thanks for your information.

I have added a str_replace in PT includes/admin.php in * apply standard filters * before the strip_tags line and this solves the problem with the &#39 apostrophe.

I am editing the text entries in phpMyAdmin to make foreign characters and other html stuff appear correctly after strip_tags. Maybe this is a slow way to do it (?) but I guess I can avoid duplicate content filters this way, too?

Best regards,
Bangkok Bob

Hi Bob, You're right about

Submitted by support on Thu, 2007-11-29 09:38

Hi Bob,

You're right about the filters. Remember that you can add as many str_replace() lines as you want to make other changes during import if that will save you editing the database directly...

Cheers,
David.