Understanding text encoding for Properties Files: Latin1 vs. UTF-8

Folks often want to have readable properties files that look like:

textInEng=this a sample of the japanese language's character set
textInJap=日本語の文字は、このサンプルでは、設定

But anyone who actually tries to use a human-readable file (like the one above) via Java will be surprised to see corrupted string values being loaded. The reason behind this is that by default Java's Properties class uses an input stream that has Latin-1 encoding. Also known as ISO-8859-1 encoding. So, reading UTF-8 as Latin-1 yields seemingly corrupted text as the boundaries for bytes that make upa character are all wrong and when Java once again converts it to UTF-8 (which is how the String class stores its data internally), instead of having access to the raw bytes, the boundaries set by Latin-1 characters are honored and that doesn't help us see any meaningful data either. This is why most properties files end up looking more like this:

textInEng=this a sample of the japanese language's character set
textInJap=\u65E5\u672C\u8A9E\u306E\u6587\u5B57\u306F\u3001\u3053\u306E\u30B5\u30F3\u30D7\u30EB\u3067\u306F\u3001\u8A2D\u5B9A

The patterns you see above (\uXXXX)are called unicode-escapes and allow a Latin-1 stream to properly convert each character into its UTF-8 equivalent and hand it off to Java whose internal storage encoding for the String class happens to be UTF-8 already.

But what if you wanted more? What if you absolutely must have human-readable properties files and they must be loaded into Java correctly?

• Well if you are using Spring, there is an easy way out: just have a look at ReloadableResourceBundleMessageSource and use it to your advantage, problem solved!
  
• Otherwise here's how you can get the best of both worlds yourself:
  
  1. Let Java's Properties class read in the UTF-8 values incorrectly with the Latin-1 encoding.
     
  2. Then go over each value and convert it back into bytes by telling the code to pass it through Latin-1 for the reverse transformation.
     
  3. Now that you have the raw-bytes again, you can read them in properly with the appropriate UTF-8 encoding!
     
  4. Set them back into your Properties object and you are good to go!
     
  5. See the code (+/-).
     
         private static void FourthTry() throws IOException {
             FileInputStream inputStream = null;
             try {
                 inputStream = new FileInputStream("utf8.properties");
                 Properties defaultProperties = new PrettyPrintProperties();
                 defaultProperties.load(inputStream);
                 for(Entry keyValuePair : defaultProperties.entrySet()) {
                     byte[] rawBytes = ((String)keyValuePair.getValue()).getBytes("ISO-8859-1");
                     String recoveredString = new String(rawBytes,"UTF-8");
                     keyValuePair.setValue(recoveredString);
                 }
                 PrintStream customConsoleOuput = new PrintStream(System.out, true, "UTF-8");
                 customConsoleOuput.println(defaultProperties.toString());
             } finally {
                 if (inputStream!= null) inputStream.close();
             }
         }
  6. If you want to test it in Eclipse, make sure to: Open Run Dialog > "your application" > Common Tab > Encoding > Other dropdown > set it to UTF-8
     
  But keep in mind, you cannot be inconsistent. You must pick choose to have the key-value pairs in your properties files be either utf8 or latin1, do NOT mix them like this:
  

  textInEng=this a sample of the japanese language's character set
  textInJap=日本語の文字は、このサンプルでは、設定
  escapedJp=\u65E5\u672C\u8A9E\u306E\u6587\u5B57\u306F\u3001\u3053\u306E\u30B5\u30F3\u30D7\u30EB\u3067\u306F\u3001\u8A2D\u5B9A

  Happy Coding!
  

How to list multiple encrypted passwords in properties files

Jasypt is an extremely useful toolkit for encrypting and decrypting passwords. It provides simple instructions for those who want to use Spring and load properties files with encrypted passwords in them.

But, what if you need to provide a whole series of usernames and passwords. How would you accomplish this?
Jasypt's out-of-the-box implementations only read one encrypted value per key=ENC(value) pair in a property file.

Here's a solution:

1. Override the EncryptablePropertyPlaceholderConfigurer's convertPropertyValue() method. Then you can split a list like the following:
   

   listOfUsernamePasswords=LIST(username1,ENC(encryptedPassword1),username2,ENC(encryptedPassword2))

   into separate values and only send the encrypted values to the parent convertPropertyValue() implementation to get the decrypted values.
   
2. Sew the list back together as a string and return a value that now reads:
   

   username1,password1,username2,password2

3. You may be wondering as to who will do the work of breaking up these comma-separated values when they get to your spring bean? The answer is simple: Spring will! For ex: If you had configured you spring bean like so:
   

   <bean id="someBean" class="com.MyBean">
       <property name="multipleUsernamePasswords" value="${listOfUsernamePasswords}" />
   </bean>

   and the multipleUsernamePasswords property is of type String[] in MyBean class, then StringArrayPropertyEditor will be used implicitly, which will break it up into a string array auto-magically. If you are interested, you can read this blog about how various structures interpret property values for Spring.
   
   
4. Now its up to use that String[] to your advantage in your code.