Articles http://dev.nirmalya.net/articles Random articles to help others who might be facing similar situations/problems (previously faced by me) during development. Sat, 12 May 2012 19:20:51 +0000 en hourly 1 Writing Own N-Grams Analyzer Vs Using ShingleAnalyzerWrapper http://dev.nirmalya.net/articles/java/writing-own-n-grams-analyzer-vs-using-shingleanalyzerwrapper http://dev.nirmalya.net/articles/java/writing-own-n-grams-analyzer-vs-using-shingleanalyzerwrapper#comments Sun, 29 Apr 2012 14:02:41 +0000 nirmalya http://dev.nirmalya.net/articles/?p=503 When extracting word-level N-grams, subclassing Apache Lucene’s Analyzer seems (at first glance) a better choice than using their ShingleAnalyzerWrapper.

Lets say you wanted to extract word-level N-Grams for the following quote, “The brick walls are there for a reason. The brick walls are not there to keep us out; the brick walls are there to give us a chance to show how badly we want something.” (by Randy Pausch).

Using the ShingleAnalyzerWrapper,

boolean outputUnigrams = false, outputUnigramsIfNoShingles = false;
int minShingleSize = 2, maxShingleSize = 2;
ShingleAnalyzerWrapper wrapper = new ShingleAnalyzerWrapper(
    new StandardAnalyzer(Version.LUCENE_36), 
    minShingleSize, maxShingleSize, tokenSeparator, 
    outputUnigrams, outputUnigramsIfNoShingles);
TokenStream tokenStream = wrapper.tokenStream("contents", new StringReader(text));

The code above yields 20 word-level 2-grams : _ brick, brick walls, walls _, _ reason, reason _, _ keep, keep us, us out, out _, _ give, give us, us _, _ chance, chance _, _ show, show how, how badly, badly we, we want, want something.

Notice the (unwanted) underscores.

Using own subclass of Apache Lucene’s Analyzer, (let’s call it) NGramAnalyzer,

public class NGramAnalyzer extends Analyzer {
  public TokenStream tokenStream(String fieldName, Reader reader) {
    Version v = Version.LUCENE_36;
    Tokenizer tokenizer = (this.convertToLowerCase 
      ? (new LowerCaseTokenizer(v, reader))
      : (new StandardTokenizer(v, reader)) );
 
    ShingleFilter ts = new ShingleFilter(tokenizer);
    ts.setOutputUnigrams(false);
    return (new StopFilter(v, ts, StopAnalyzer.ENGLISH_STOP_WORDS_SET));
  }
}

The code above yields 26 word-level 2-grams : the brick, brick walls, walls are, are there, there for, for a, a reason, reason the, are not, not there, there to, to keep, keep us, us out, out the, to give, give us, us a, a chance, chance to, to show, show how, how badly, badly we, we want, want something.

Notice the inclusion of 2-grams such as “are there“, “a reason“, “reason the” which are either missing or partially substituted by underscores in the 2-grams output by the ShingleAnalyzerWrapper (because they contain stop words which are part of the ENGLISH_STOP_WORDS_SET).

So, it might be better to use the former (ShingleAnalyzerWrapper), with minor modifications,

CharTermAttribute termAttribute = 
  (CharTermAttribute) tokenStream.getAttribute(CharTermAttribute.class);
LinkedHashSet<String> tokens = new LinkedHashSet<String>();
char underscore = "_".charAt(0);
try {
  while (stream.incrementToken()) {
    String n_gram = termAttribute.toString();
    int ln = n_gram.length();
    if ((n_gram.charAt(0) == underscore)) {
      if (outputUnigrams) { tokens.add(n_gram.substring(2)); }
    }
    else if ((n_gram.charAt(ln - 1) == underscore)) {
      if (outputUnigrams) { tokens.add(n_gram.substring(0, ln - 2)); }
    }
    else
      tokens.add(n_gram);
  }
}
catch (IOException e) {
  // .. handle exception
}

which now yields 9 word-level 2-grams, brick walls, keep us, us out, give us, show how, how badly, badly we, we want, want something.

Hope this helps.

]]> http://dev.nirmalya.net/articles/java/writing-own-n-grams-analyzer-vs-using-shingleanalyzerwrapper/feed 0 Equivalent Of “Like” For MongoDB http://dev.nirmalya.net/articles/java/equivalent-of-like-for-mongodb http://dev.nirmalya.net/articles/java/equivalent-of-like-for-mongodb#comments Sun, 25 Mar 2012 09:10:47 +0000 nirmalya http://dev.nirmalya.net/articles/?p=493 Command db.foo.find({"firstName":/ss/}) on the Mongo shell returns results similar to

{ "firstName" : "Issac", "lastName" : "Newton", "yearOfBirth" : 1642 }

Repeating the same using Spring Data for MongoDB did not go smoothly. I kept encountering JSON parse exceptions for the JSON string {"firstName":/ss/}.

Referring to MongoDB manual, the $regex clause should help. Spring Data’s Criteria class has 2 related methods (Criteria.regex(String) and Criteria.regex(String, String)). However, I was unable to get them to work either. So I resorted to using the MongoDB Java Driver directly. A snippet of the working code,

  public List<Person> findPeopleHavingFirstNameContaining(String searchString) {
    List<Person> people = null;
    Mongo m = null;
    try {
      m = new Mongo("localhost", 27017);// simplified for this post
      DB db = m.getDB("test");
      BasicDBObject query = new BasicDBObject();
      DBCollection coll = db.getCollection("foo");
      query.put("firstName", 
        new BasicDBObject("$regex", String.format(".*((?i)%s).*", searchString)) );
      DBCursor cur = coll.find(query);
      int numRecords = (cur != null) ? cur.count() : 0;
      if (numRecords > 0) {
        people = new ArrayList<Person>();
        while (cur.hasNext()) {
          DBObject o = cur.next();
          people.add(new Person(
                       (String) o.get("firstName"), 
                       (String) o.get("lastName"), 
                       ((Integer) o.get("yearOfBirth")).intValue()
                    ));
        }
      }
    }
    catch (Exception e) { /* handle exception .. */ }
    finally { if (m != null) { m.close(); } }
    return people;
  }

And here is the domain object,

public class Person {
  private String firstName, lastName;
  private int yearOfBirth;
  public Person(String fName, String lName, int year) {
    setFirstName(fName);
    setLastName(lName);
    setYearOfBirth(year);
  }
  // getter/setter methods follow ..
}
]]> http://dev.nirmalya.net/articles/java/equivalent-of-like-for-mongodb/feed 0 Quickly Debugging Your Hadoop Program http://dev.nirmalya.net/articles/java/quickly-debugging-your-hadoop-program http://dev.nirmalya.net/articles/java/quickly-debugging-your-hadoop-program#comments Thu, 15 Mar 2012 10:30:34 +0000 nirmalya http://dev.nirmalya.net/articles/?p=505 This quick tip applies to situations when you are trying to debug your Hadoop program and the log messages (or System.outs) are not appearing on file or console.

Edit the /usr/local/hadoop/conf/log4j.properties file (path relevant to the vanilla Apache Hadoop distribution). Update the log4j.rootLogger to have console appended to it, similar to:

log4j.rootLogger=${hadoop.root.logger}, EventCounter, console

The log files are under subdirectories under /usr/local/hadoop/logs/userlogs similar to job_201203151712_000x. Select the most recent (the one with largest x).

Under /usr/local/hadoop/logs/userlogs there are subdirectories similar to

attempt_201203151712_0008_m_000000_0  attempt_201203151712_0008_m_000002_0  
attempt_201203151712_0008_r_000000_0  attempt_201203151712_0008_m_000001_0
attempt_201203151712_0008_m_000003_0  job-acls.xml

The naming convention is pretty obvious, i.e. the _m_ ones are for the Mappers, _r_ for Reducers. Each of these _m_/_r_ subdirectories have 4 files,

log.index  stderr  stdout  syslog

Your log messages appear inside the stderr file.
Your System.outs appear inside the stdout file.

]]> http://dev.nirmalya.net/articles/java/quickly-debugging-your-hadoop-program/feed 0 Social Media Counter Using MongoDB And Spring Data http://dev.nirmalya.net/articles/java/social-media-counter-using-mongodb-and-spring-data http://dev.nirmalya.net/articles/java/social-media-counter-using-mongodb-and-spring-data#comments Thu, 23 Feb 2012 11:30:52 +0000 nirmalya http://dev.nirmalya.net/articles/?p=406 About a month ago, I created a simple project to better manage reading list (fed by an ever growing number of RSS feeds) by prioritizing which articles to read based on attributes such as URLs’ popularity, age, tags, etc..

The popularity of an URL can be determined based on its social media counters (Facebook likes, Tweet count, Google +1′s, LinkedIn shares, etc.), number of comments, etc.

Information stored for one URL is likely to be very different from that stored for another. For example,

  • URL1 only has a Tweet count and the number of LinkedIn shares,
  • URL2 has Facebook likes, Tweet count, LinkedIn shares and a set of tags
  • URL3 has Facebook likes, Tweet count, LinkedIn shares and Google +1′s (introduced later in the project),

MongoDB is excellent for handling schemaless data similar to the simple example above. Spring Data makes it very easy to integrate MongoDB into a new/existing project. In this article, I describe the project’s setup, the relevant sections of the project’s pom.xml, part of the DAO code and a simple aggregation done by making use of MongoDB’s Map Reduce feature. Specifics of trending algorithms, the UIs will be covered in later articles.

Project Directory Layout
I have used the standard directory layout common to most Maven projects. I have the used the common DAO pattern which allows replacing of the actual persitence layer (in this example, MongoDB) with any other NoSQL or relational database. Code responsible for the social media counters (such as the FacebookWorker, GooglePlusOneWorker, TwitterWorker, etc.) are grouped under the worker package. Code for parsing RSS feeds is based on http://www.vogella.de/articles/RSSFeed/article.html.

¦ pom.xml
+---src
¦ +---main
¦ ¦ +---java
¦ ¦ ¦ +---de
¦ ¦ ¦ ¦ +---vogella
¦ ¦ ¦ ¦ +---rss
¦ ¦ ¦ +---net
¦ ¦ ¦ +---nirmalya
¦ ¦ ¦ +---urlpopularitycounter
¦ ¦ ¦ +---dao
¦ ¦ ¦ ¦ +---mongo
¦ ¦ ¦ +---worker
¦ ¦ +---resources
¦ ¦ ------applicationContext.xml
¦ ¦ ------quartz.properties
¦ +---test
¦ +---java
¦ ¦ +---dev
¦ ¦ +---nirmalya
¦ ¦ +---net
¦ ¦ +---urlpopularitycounter
¦ +---resources

pom.xml
Pasted below are relevant sections of the project’s pom.xml to get started with Spring Data and MongoDB. First, the repositories section,

  <repositories>
    <repository>
      <id>spring-milestone</id>
      <name>Spring Maven MILESTONE Repository</name>
      <url>http://maven.springframework.org/milestone</url>
    </repository>
  </repositories>

Relevant parts of the dependencies section,

    <dependency>
      <groupId>commons-collections</groupId>
      <artifactId>commons-collections</artifactId>
      <version>3.2.1</version>
    </dependency>
    <dependency>
      <groupId>commons-logging</groupId>
      <artifactId>commons-logging</artifactId>
      <version>1.1.1</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-core</artifactId>
      <version>3.0.6.RELEASE</version>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-context-support</artifactId>
      <version>3.1.0.RELEASE</version>
    </dependency>
    <dependency>
      <groupId>org.springframework.data</groupId>
      <artifactId>spring-data-mongodb</artifactId>
      <version>1.0.1.RELEASE</version>
    </dependency>

I used JsonPath (think of it as XPath for JSON) to simplify parsing the JSON strings returned by some of the social media counters. Relevant dependency,

    <dependency>
      <groupId>com.jayway.jsonpath</groupId>
      <artifactId>json-path</artifactId>
      <version>0.5.5</version>
    </dependency>

Relevant dependency for the RSS feed parsing code,

    <dependency>
      <groupId>javax.xml.stream</groupId>
      <artifactId>stax-api</artifactId>
      <version>1.0-2</version>
    </dependency>
    <dependency>
      <groupId>org.json</groupId>
      <artifactId>json</artifactId>
      <version>20090211</version>
    </dependency>

I used the Quartz Scheduler for scheduling the RSS feed checking jobs. Relevant dependency

    <dependency>
      <groupId>org.opensymphony.quartz</groupId>
      <artifactId>quartz</artifactId>
      <version>1.6.1</version>
    </dependency>

applicationContext.xml
Relevant sections of the Spring configuration file,

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xmlns:context="http://www.springframework.org/schema/context"
          xmlns:mongo="http://www.springframework.org/schema/data/mongo"
          xmlns:util="http://www.springframework.org/schema/util"
          xsi:schemaLocation=
          "http://www.springframework.org/schema/context
          http://www.springframework.org/schema/context/spring-context-3.0.xsd
          http://www.springframework.org/schema/data/mongo
          http://www.springframework.org/schema/data/mongo/spring-mongo-1.0.xsd
          http://www.springframework.org/schema/beans
          http://www.springframework.org/schema/beans/spring-beans-3.0.xsd"
          >
 
    <context:annotation-config/>
    <mongo:mongo host="127.0.0.1" port="27017"/>
    <bean id="mongoTemplate" 
          class="org.springframework.data.mongodb.core.MongoTemplate">
      <constructor-arg ref="mongo"/>
      <constructor-arg name="databaseName" value="test"/>
    </bean>
 
    <bean id="urlStatsDao" 
          class="net.nirmalya.urlpopularitycounter.dao.mongo.UrlStatsDaoImpl"
          autowire="byType"/>
 
    <!-- other bean definitions omitted for sake of brevity -->
</beans>

Some Code
The DAO,

public interface UrlStatsDao {
  public UrlStats find(String url);
  public List<UrlStats> findLeastPopular(int n);
  public List<UrlStats> findMostPopular(int n);
  public List<UrlStats> findMostRecent(int n);
  public List<UrlStats> findOldest(int n);
  public void insert(UrlStats urlStats);
  public void insertOrUpdate(UrlStats urlStats);
  public void remove(String url);
  public Map<Integer, Integer> retrieveNumUrlsGroupBy(GroupByTime groupByTime);
  // .. other methods 
}

Partial implementation of UrlStatsDaoImpl,

import static org.springframework.data.mongodb.core.query.Criteria.where;
import java.util.Date;
import java.util.List;
import net.nirmalya.urlpopularitycounter.UrlStats;
import net.nirmalya.urlpopularitycounter.dao.UrlStatsDao;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.mongodb.core.MongoOperations;
import org.springframework.data.mongodb.core.query.Order;
import org.springframework.data.mongodb.core.query.Query;
import org.springframework.data.mongodb.core.query.Sort;
import org.springframework.data.mongodb.core.query.Update;
 
public class UrlStatsDaoImpl implements UrlStatsDao {
 
  // .. other methods omitted for sake of brevity 
 
    public void insertOrUpdate(UrlStats urlStats) {
        Update u = update("facebookComments", urlStats.getFacebookComments(), null);
        u = update("facebookShares", urlStats.getFacebookShares(), u);
        u = update("googlePlusOneCount", urlStats.getGooglePlusOneCount(), u);
        u = update("linkedInCount", urlStats.getLinkedInCount(), u);
        u = update("twitterCount", urlStats.getTwitterCount(), u);
        u = update("total", urlStats.getTotal(), u);
        u = update("dateAdded", urlStats.getDateAdded(), u);
        if (urlStats.getTotal() > 0) {
            u.addToSet("totalHistory", urlStats.getTotal());
        }
        upsert(urlStats, u);
    }
 
    private Update update(String key, Object v, Update u) {
        if (v == null
                || (v.getClass() == Integer.class && ((Integer) v).intValue() > 0)
                || (v.getClass() == Date.class)) {
            if (u == null) {
                u = new Update();
            }
            u.set(key, v);
        }
        return u;
    }
 
    private void upsert(UrlStats urlStats, Update u) {
        if (u != null) {
            Date lastUpdated = new Date();
            u.set("lastUpdated", lastUpdated);
            u.addToSet("lastUpdatedHistory", lastUpdated);
            mongoTemplate.upsert(getQuery(urlStats.getUrl()), u, UrlStats.class);
        } else {
            mongoTemplate.save(urlStats, collectionName);
        }
    }
 
}

Partial implementation of UrlStatsWorker,

public class UrlStatsWorker implements Runnable {
    private volatile boolean stopRequested = false;
    private int threadPoolSize = 4;
    private final Log logger = LogFactory.getLog(UrlStatsWorker.class);
    private UrlStatsDao urlStatsDao;
 
  // .. other methods omitted for sake of brevity 
 
    private void updateStats(String trackedUrl, ExecutorService execSrvc) {
        LinkedHashMap<String, Integer> counts = new LinkedHashMap<String, Integer>();
        try {
            Collection<Callable<Integer>> tasks = 
              new LinkedList<Callable<Integer>>();
            String[] names = 
              { "FaceBookShares", "GooglePlusOneCount", 
                "LinkedInCount", "TwitterCount" };
            tasks.add(new FacebookWorker(trackedUrl));
            tasks.add(new GooglePlusOneWorker(trackedUrl));
            tasks.add(new LinkedInWorker(trackedUrl));
            tasks.add(new TwitterWorker(trackedUrl));
            List<Future<Integer>> list = execSrvc.invokeAll(tasks);
            int index = 0;
            for (Future<Integer> f : list) {
                counts.put(names[index++], f.get());
            }
        } catch (Exception e) {
            logger.info(e);
        }
 
        // Saving / updating ..
        UrlStats urlStatsB4 = urlStatsDao.find(trackedUrl);
        UrlStats urlStatsAfter = new UrlStats(trackedUrl);
        urlStatsAfter.setFacebookComments(counts.get("FaceBookShares"));
        urlStatsAfter.setGooglePlusOneCount(counts.get("GooglePlusOneCount"));
        urlStatsAfter.setLinkedInCount(counts.get("LinkedInCount"));
        urlStatsAfter.setTwitterCount(counts.get("TwitterCount"));
        urlStatsAfter.setDateAdded(urlStatsB4.getDateAdded());
        if (urlStatsAfter.getDateAdded() == null) {
            urlStatsAfter.setDateAdded(new Date());
        }
        urlStatsDao.insertOrUpdate(urlStatsAfter);
    }

Partial implementation of one of the social media counter classes (GooglePlusOneWorker),

class GooglePlusOneWorker implements Callable<Integer> {
    private final Log logger = LogFactory.getLog(GooglePlusOneWorker.class);
    private String trackedUrl;
 
    GooglePlusOneWorker(String trackedUrl) {
        this.trackedUrl = trackedUrl;
    }
 
    public Integer call() throws Exception {
        int count = 0;
        String urlForStats = " .. ";
        String data = " .. ";
        try {
            // Do a HTTP Post ..
            // Read the reply from the server
            // Parse the JSON string returned - use JsonPath.read(jsonString, "path")
            Double countd = JsonPath.read(replyFromServer, ".. xpath ..");
            count = countd.intValue();
        } catch (Exception e) {
            logger.warn(e);
        }
        return count;
    }
}

The representation of the “document” (think of it as a table record for those new to NoSQL)

public class UrlStats {
    private Date dateAdded, lastUpdated;
    private int facebookComments, facebookShares, linkedInCount,
      googlePlusOneCount, total, twitterCount;
    private String url, urlTitle;  
  // .. getter/setter methods omitted for sake of brevity

“Documents”
Command db.urlStats.find({}).limit(2).pretty(); on the Mongo shell returns results similar to

{
    "_id" : ObjectId("4f39bf6af466eb9eb443e3cb"),
    "dateAdded" : ISODate("2012-02-14T01:56:58.321Z"),
    "facebookComments" : 2196,
    "googlePlusOneCount" : 36,
    "lastUpdated" : ISODate("2012-02-23T06:27:34.651Z"),
    "lastUpdatedHistory" : [
        ISODate("2012-02-14T01:57:01.178Z"), ISODate("2012-02-15T09:39:01.981Z"),
        ISODate("2012-02-16T08:02:31.182Z"), ISODate("2012-02-17T10:34:31.260Z"),
        ISODate("2012-02-18T16:18:35.770Z"), ISODate("2012-02-19T21:40:32.719Z"),
        ISODate("2012-02-20T00:06:32.823Z"), ISODate("2012-02-21T23:31:30.684Z"),
        ISODate("2012-02-22T20:54:32.098Z"), ISODate("2012-02-23T06:27:34.651Z")
    ],
    "linkedInCount" : 163,
    "total" : 4258,
    "totalHistory" : [ 4119, 4203, 4223, 4240, 4250, 4254, 4255, 4256, 4257, 4258 ],
    "twitterCount" : 1863,
    "url" : "http://www.bbc.co.uk/nature/16944753",
    "urlTitle" : "Why Zebra evolved stripes?"
},
{
    "_id" : ObjectId("4f32408d88d848ee8340aaed"),
    "dateAdded" : ISODate("2012-02-01T00:00:00Z"),
    "facebookComments" : 31,
    "googlePlusOneCount" : 19,
    "lastUpdated" : ISODate("2012-02-11T20:08:37.609Z"),
    "lastUpdatedHistory" : [
        ISODate("2012-02-08T09:30:01.186Z"), ISODate("2012-02-09T10:29:04.646Z"),
        ISODate("2012-02-10T09:43:31.200Z"), ISODate("2012-02-11T20:08:37.609Z")
    ],
    "linkedInCount" : 10,
    "total" : 240,
    "totalHistory" : [ 221, 229, 230, 240 ],
    "twitterCount" : 180,
    "url" : "http://www.engadget.com/2012/02/01/german-court-denies-apple-request-for-preliminary-ban-on-galaxy/",
    "urlTitle" : "German court denies Apple request for preliminary ban on Galaxy Tab 10.1N and Galaxy Nexus sales"
}

Map Reduce

Using MongoDB’s Map Reduce feature to count number of documents grouped by a unit of time (years, months, days, hours, minutes, seconds, milliseconds). Added the following method to the UrlStatsDaoImpl,

    public Map<Integer, Integer> retrieveNumUrlsGroupBy(GroupByTime groupByTime) {
        String mapFunction = "function() { " +
          "var ts_x = this.ts." + groupByTime.getMethodName() + ";" + 
          "emit(ts_x, 1); }";
        String reduceFunction = "function(key, values) { " +
          " var total = 0; " + 
          " for(var i = 0; i < values.length; i++) { " + 
          "   total += values[i]; } " +
          " return total; }";
 
        TreeMap<Integer, Integer> mrResults = new TreeMap<Integer, Integer>();
 
        @SuppressWarnings("rawtypes")
        MapReduceResults<Map> results = mongoTemplate.mapReduce(
          collectionName, 
          mapFunction, reduceFunction,
          Map.class);
        if (results.getRawResults() != null) {
            Object obj = results.getRawResults().get("results");
            if (obj != null) {
                BasicDBList list = (BasicDBList) obj;
                Iterator<Object> it = list.iterator();
                while (it.hasNext()) {
                    BasicDBObject lItem = (BasicDBObject) it.next();
                    @SuppressWarnings("unchecked")
                    Map<String, Double> lItemMap = lItem.toMap();
                    if (lItemMap != null) {
                        mrResults.put(lItemMap.get("_id").intValue(), 
                          lItemMap.get("value").intValue());
                    }
                }
            }
        }
 
        return mrResults;
    }

and added a new Enum GroupByTime,

public enum GroupByTime {
    YEARS("getYear()"), MONTHS("getMonth()"), DAYS("getDate()"), HOURS("getHours()");
    private String methodName;
 
    GroupByTime(String methodNameSpecified) {
        this.methodName = methodNameSpecified;
    }
 
    public String getMethodName() {
        return methodName;
    }
}

A simple JUnit test helps verify

    @Test
    public final void testRetrieveNumUrlsGroupByXYZ() {
        ApplicationContext ctx = 
          new GenericXmlApplicationContext("classpath:/applicationContext.xml");
        UrlStatsDao dao = 
          (UrlStatsDao) ((ctx!= null) ? ctx.getBean("urlStatsDao") : null);
        assertNotNull(dao);
        Map<Integer, Integer> numUrlsGroupedByDays = 
          dao.retrieveNumUrlsGroupBy(GroupByTime.DAYS);
        assertNotNull(numUrlsGroupedByDays);
        System.out.println("numUrlsGroupedByDays = " + numUrlsGroupedByDays);
    }

The Map Reduce results look similar to {1=1, 2=21, 8=788, .. , 21=272, 22=299, 23=68} where the keys represent the day of the month and the number of URLs added on that particular day.

Hope the snippets helps you in your project.

References

[Update #1, Date : 15 April 2012]
Previously spring-data-mongodb version 1.0.0.RC1 worked with spring-context-support version 3.0.6.RELEASE. It should now be spring-data-mongodb version 1.0.1.RELEASE with spring-context-support version 3.1.0.RELEASE.

]]> http://dev.nirmalya.net/articles/java/social-media-counter-using-mongodb-and-spring-data/feed 0 Installing Tomcat 7 On Ubuntu And Accessing It Remotely http://dev.nirmalya.net/articles/setup/tomcat-7-ubuntu-remote-access http://dev.nirmalya.net/articles/setup/tomcat-7-ubuntu-remote-access#comments Wed, 22 Feb 2012 07:23:24 +0000 nirmalya http://dev.nirmalya.net/articles/?p=364 I installed Tomcat 7.0.26 on a standard.xsmall instance with a Ubuntu 11.04 image following some pretty good instructions on Diego Benna’s blog. After updating the required configuration files, I was able to access the server locally, but unable to access remotely. For future reference, listed below are steps I followed to resolve it.

Installing Java 7
Please refer to the excellent instructions at http://brunoreis.com/tech/intalling-java-ubuntu-natty/.

Installing Tomcat 7.0.26
Preparation

$ tar xvzf apache-tomcat-7.0.26.tar.gz
$ sudo mv apache-tomcat-7.0.26/ /usr/share/tomcat7

Next, edit /etc/environment. Contents of this file

JAVA_HOME="/usr/lib/jvm/java-7-sun/"
JRE_HOME="/usr/lib/jvm/java-7-sun/jre"
PATH="/usr/local/sbin:/usr/games:$JAVA_HOME:$JRE_HOME"

Next, edit /usr/share/tomcat7/bin/catalina.sh. Add in these 2 lines just after the #!/bin/sh

JAVA_HOME="/usr/lib/jvm/java-7-sun/"
JRE_HOME="/usr/lib/jvm/java-7-sun/jre"

Next, edit /usr/share/tomcat7/conf/tomcat-users.xml.
Add the following just before the end of the tomcat-users element.

<user username="yourAdminUsername" password="yourAdminPassword"
 roles="manager-gui,admin-gui,manager,admin,manager-script,admin-script"/>

Starting up Tomcat

$ sudo /usr/share/tomcat7/bin/startup.sh

Since I do not have local access to my instance, I used wget to check the local access.

$ cd /tmp
$ wget http://127.0.0.1:8080

The contents are written to index.html. You can then try to wget http://your-instance-public-ip:8080, but most likely it will not work at this point.

Configuring The Firewall
Please refer to the excellent instructions at http://chiralsoftware.com/linux-system-administration/ubuntu-firewall-iptables.seam. Summarized below

$ iptables -I INPUT 1 -i lo -j ACCEPT
$ iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
$ iptables -A INPUT -p tcp --dport 80 -j ACCEPT
$ iptables -A INPUT -p tcp --dport 22 -j ACCEPT
$ iptables -A INPUT -j DROP
$ iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080
$ iptables -I INPUT 3 -p tcp --dport 8080 -j ACCEPT
$ iptables -t nat -A OUTPUT -o lo -p tcp --dport 80 -j REDIRECT --to-port 8080
$ iptables-save > /etc/iptables.rules

Next, reboot your instance, restart Tomcat and you should now be able to access http://your-instance-public-ip

Hope this helps.

References

]]> http://dev.nirmalya.net/articles/setup/tomcat-7-ubuntu-remote-access/feed 1 SimpleJPA + Maven + Dependencies http://dev.nirmalya.net/articles/java/simplejpa-maven-dependencies http://dev.nirmalya.net/articles/java/simplejpa-maven-dependencies#comments Sun, 01 Jan 2012 19:44:28 +0000 nirmalya http://dev.nirmalya.net/articles/?p=342 Whilst I am very grateful to the folks who developed SimpleJPA, I found it a bit problematic building (with Maven) a project having SimpleJPA as of its dependencies. In this article, I attempt to make it easier for others who intend to use Maven for building their project which uses SimpleJPA.


The Need To Add SimpleJPA To Maven Local Repository
To begin with, (as of now) one cannot just add in a dependency similar to the one below and expect the build to proceed smoothly.

    <dependency>
      <groupId>com.spaceprogram</groupId>
      <artifactId>simplejpa</artifactId>
      <version>1.5</version>
    </dependency>

I had to download the SimpleJPA JAR file and it to my maven local repository. To do so, you may use the command,

mvn install:install-file
 -Dfile=PathToJARFile\simplejpa-1.5.jar
 -DgroupId=com.spaceprogram
 -DartifactId=simplejpa -Dversion=1.5 -Dpackaging=jar -DgeneratePom=true

The Need To Manually Manage SimpleJPA’s Dependencies
SimpleJPA’sGettingStarted Wiki page lists its dependencies. Since I couldn’t find a working Maven pom.xml for SimpleJPA, I created one and pasted below is the dependencies section which you can just copy into your project’s pom.xml.

 <dependencies>
    <dependency>
      <groupId>cglib</groupId>
      <artifactId>cglib-nodep</artifactId>
      <version>2.2.2</version>
    </dependency>
    <dependency>
      <groupId>com.amazonaws</groupId>
      <artifactId>aws-java-sdk</artifactId>
      <version>1.0.002</version>
    </dependency>
    <dependency>
      <groupId>com.spaceprogram</groupId>
      <artifactId>kitty-cache</artifactId>
      <version>1.2</version>
    </dependency>
    <dependency>
      <groupId>commons-codec</groupId>
      <artifactId>commons-codec</artifactId>
      <version>1.6</version>
    </dependency>
    <dependency>
      <groupId>commons-collections</groupId>
      <artifactId>commons-collections</artifactId>
      <version>3.2.1</version>
    </dependency>
    <dependency>
      <groupId>commons-lang</groupId>
      <artifactId>commons-lang</artifactId>
      <version>2.6</version>
    </dependency>
    <dependency>
      <groupId>commons-logging</groupId>
      <artifactId>commons-logging</artifactId>
      <version>1.1.1</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>net.sf.ehcache</groupId>
      <artifactId>ehcache-core</artifactId>
      <version>2.5.0</version>
    </dependency>
    <dependency>
      <groupId>net.sf.scannotation</groupId>
      <artifactId>scannotation</artifactId>
      <version>1.0.2</version>
    </dependency>
    <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpclient</artifactId>
      <version>4.1.2</version>
    </dependency>
    <dependency>
      <groupId>org.hibernate</groupId>
      <artifactId>ejb3-persistence</artifactId>
      <version>1.0.2.GA</version>
    </dependency>
    <dependency>
      <groupId>org.javassist</groupId>
      <artifactId>javassist</artifactId>
      <version>3.15.0-GA</version>
    </dependency>
    <!-- SimpleJPA -->
    <dependency>
      <groupId>com.spaceprogram</groupId>
      <artifactId>simplejpa</artifactId>
      <version>1.5</version>
    </dependency>
  </dependencies>

Please take note

  • GroupIds, ArtifcatIds, versions determined using http://search.maven.org/#search
  • You will also need to download ejb3-persistence-1.0.2.GA.jar from here and add to maven local repository using the command 
    mvn install:install-file
 -Dfile=PathToJARFile\ejb3-persistence-1.0.2.GA.jar
 -DgroupId=org.hibernate
 -DartifactId=ejb3-persistence
 -Dversion=1.0.2.GA -Dpackaging=jar
 -DgeneratePom=false
  • You will also need to download KittyCache-1.2.jar from here and add to maven local repository using the command 
    mvn install:install-file
 -Dfile=PathToJARFile\KittyCache-1.2.jar
 -DgroupId=com.spaceprogram
 -DartifactId=kitty-cache
 -Dversion=1.2 -Dpackaging=jar -DgeneratePom=true

References:

]]> http://dev.nirmalya.net/articles/java/simplejpa-maven-dependencies/feed 0 Regex Pattern In XSD For Comma-Separated Attribute Value http://dev.nirmalya.net/articles/uncategorized/regex-pattern-in-xsd-for-comma-separated-attribute-value http://dev.nirmalya.net/articles/uncategorized/regex-pattern-in-xsd-for-comma-separated-attribute-value#comments Wed, 16 Jun 2010 07:40:14 +0000 nirmalya http://dev.nirmalya.net/articles/?p=338 Relevant Section Of The XML

<my_element xyz="ABC3"/>
<my_element xyz="ABC21,a1"/>

Relevant Section Of The XSD

1
2
3
4
5
6
7
8
9
10
11

<xs:element name="my_element" maxOccurs="unbounded" minOccurs="1">
  <xs:complexType>
    <xs:attribute name="xyz" use="required">
      <xs:simpleType> 
        <xs:restriction base="xs:string"> 
          <xs:pattern value="[a-zA-Z0-9]+[,a-zA-Z0-9]*"/>
        </xs:restriction> 
      </xs:simpleType>
    </xs:attribute>
  </xs:complexType>
</xs:element>
]]> http://dev.nirmalya.net/articles/uncategorized/regex-pattern-in-xsd-for-comma-separated-attribute-value/feed 0 Chaining 2 jQuery UI Datepickers http://dev.nirmalya.net/articles/javascript/chaining-2-jquery-ui-datepickers http://dev.nirmalya.net/articles/javascript/chaining-2-jquery-ui-datepickers#comments Thu, 10 Jun 2010 07:49:51 +0000 nirmalya http://dev.nirmalya.net/articles/?p=331 A simple example to chain 2 jQuery UI datepickers such that when a date is selected in the first datepicker (a.k.a. the ‘from date‘, date1), the minimum date in the second datepicker (a.k.a. the ‘to date‘, date2) is set accordingly. Similarly it handles the case where the ‘to date‘ is selected before the ‘from date‘, in which case the maximum date in the ‘from date‘ is lesser than the ‘to date‘ but before current date.

Relevant HTML For Datepicker Placeholders

<p><input type="text" name="date1" id="date1" /></p>
<p><input type="text" name="date2" id="date2" /></p>

Relevant jQuery Setup Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

<script>
  $(document).ready(function() {  
    $('#date1').datepicker({
      autoSize: true, 
      constrainInput: true, 
      dateFormat: 'ddMyy', 
      minDate: 0, 
      maxDate: '+90D', 
      onSelect: function(dateText, inst) { 
        $('#date2').datepicker("option", "minDate",dateText); 
      } 
    });
    $('#date2').datepicker({
      autoSize: true, 
      constrainInput: true, 
      dateFormat: 'ddMyy', 
      minDate: 0, 
      maxDate: '+90D', 
      onSelect: function(dateText, inst) { 
        $('#date1').datepicker("option", "maxDate",dateText); 
      } 
    });
    //Other setup
  });
</script>

In the jQuery setup code above, I have also set the following

  • Do not allow selection of dates before current date (minDate: 0)
  • Do not allow selection of dates 90 days after current date (maxDate: '+90D')
  • Date format similar to ‘17Jun2010‘
  • The ‘constrainInput: true‘ indicates that the input field is constrained to those characters allowed by the current dateFormat.
]]> http://dev.nirmalya.net/articles/javascript/chaining-2-jquery-ui-datepickers/feed 4 Spring + JPA + Hibernate + Tomcat + EHCache http://dev.nirmalya.net/articles/java/spring-jpa-hibernate-tomcat-ehcache http://dev.nirmalya.net/articles/java/spring-jpa-hibernate-tomcat-ehcache#comments Sat, 01 May 2010 17:27:09 +0000 nirmalya http://dev.nirmalya.net/articles/?p=305 Having @ManyToMany(fetch=FetchType.EAGER) attributes can slow down retrieval quite significantly (up to 40 times slower).

I recall reading somewhere that FetchType.EAGER is the default for @ManyToMany associations. Also, from experience I noticed that setting FetchType.LAZY caused a org.hibernate.LazyInitializationException thrown with exception message similar to, failed to lazily initialize a collection of role: com.xyz.domain.EntityOne.images, no session or session was closed.

So, it seemed that the only other way to quickly reduce the time it took (for the retrieval) was to look into the caching options such as query caching, second-level caching, both of which are supported by Hibernate (the webapp’s JPA provider). For more information on second-level caching, please refer to this article.

Integrating Spring + JPA + Hibernate + Tomcat + EHCache took me a few hours this afternoon, but the effort paid off. The retrieval is now 40 times faster!

This post is summarizing the setup involved in getting them all to work together.

Relevant Section Of persistence.xml:

<provider>org.hibernate.ejb.HibernatePersistence</provider>
<shared-cache-mode>ENABLE_SELECTIVE</shared-cache-mode>
<properties>
  <property name="hibernate.cache.provider_class" 
    value="org.hibernate.cache.SingletonEhCacheProvider" />
  <property name="hibernate.cache.provider_configuration" value="/ehcache.xml" />
  <property name="hibernate.cache.use_second_level_cache" value="true" />
  <property name="hibernate.cache.use_query_cache" value="true" />
</properties>

Notes:

  • The persistence.xml is located at {tomcat}/webapps/{your-webapp}/META-INF.
  • I used org.hibernate.cache.SingletonEhCacheProvider instead of org.hibernate.cache.EhCacheProvider. I was getting a WARN message if I used org.hibernate.cache.EhCacheProvider. I referred to this post for a fix.

Relevant Section Of ehcache.xml:

  <diskStore path="user.dir/mywebapp-special-cache-folder"/>
 
  <defaultCache eternal="false" overflowToDisk="false"
    maxElementsInMemory="1000" timeToIdleSeconds="30" timeToLiveSeconds="60"/>
 
  <cache name="com.xyz.domain.EntityTwo" eternal="false" overflowToDisk="true" 
    maxElementsInMemory="1000" timeToIdleSeconds="300" timeToLiveSeconds="600"
    diskPersistent="true" diskExpiryThreadIntervalSeconds="300"/>
 
  <cache name="com.xyz.domain.EntityOne" eternal="false" overflowToDisk="true" 
    maxElementsInMemory="1000" timeToIdleSeconds="300" timeToLiveSeconds="600" 
    diskPersistent="true" diskExpiryThreadIntervalSeconds="300"/>
 
  <cache name="com.xyz.domain.EntityOne.images" eternal="false" 
    overflowToDisk="true" 
    maxElementsInMemory="1000" timeToIdleSeconds="300" timeToLiveSeconds="600" 
    diskPersistent="true" diskExpiryThreadIntervalSeconds="300"/>

Notes:

  • The ehcache.xml is located at {tomcat}/webapps/{your-webapp}/WEB-INF/classes.
  • The diskStore element indicates where the files to be used for caching will be stored for the entities I wish to be made persistent to disk. In my web application, the files are stored under {tomcat}/bin/{mywebapp-special-cache-folder}
  • diskPersistent="true" indicates that the disk store (for the specific entity) is persistent between cache and VM restarts. Please refer to the EHCache documentation on disk storage for more information.

Relevant JARs: under {tomcat}/webapps/{your-webapp}/WEB-INF/lib

  • ehcache-core-2.0.1.jar
  • hibernate3.jar
  • hibernate-jpa-2.0-api-1.0.0.Final.jar
  • slf4j-api-1.5.8.jar
  • slf4j-log4j12-1.5.6.jar
  • Spring 3.0.2 JARs..

Relevant Section Of Spring Configuration:

<!-- there should be a way out of hardcoding the location of the properties file -->
<bean 
  class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
  <property name="location" value="/WEB-INF/spring-hes-db.properties" />
</bean>
 
<!-- ENTITY MANAGER FACTORY -->
<!-- LocalEntityManagerFactoryBean did not work for me -->
<bean id="emf-p" 
  class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
  <property name="persistenceUnitManager" ref="pum"/>
  <property name="persistenceUnitName" value="pu1"/>
  <property name="dataSource" ref="dataSource-p" />
  <property name="jpaVendorAdapter">
    <bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
      <property name="database" value="MYSQL" />
      <property name="showSql" value="true" />
      <property name="generateDdl" value="true" />
      <property name="databasePlatform" value="org.hibernate.dialect.MySQLDialect"/>
    </bean>
  </property>
</bean>
 
<!-- TRANSACTION MANAGER -->
<bean id="transactionManager" 
  class="org.springframework.orm.jpa.JpaTransactionManager">
  <property name="entityManagerFactory" ref="emf-p"/>
  <property name="dataSource" ref="dataSource-p"/>
</bean>
 
<!-- DATA SOURCES -->
<bean id="dataSource-p" 
    class="org.springframework.jdbc.datasource.DriverManagerDataSource">
  <property name="driverClassName" value="com.mysql.jdbc.Driver" />
  <property name="url" 
    value="jdbc:mysql://${db.host}:${db.port}/${db.name}"/>
  <property name="username" value="${db.username}" />
  <property name="password" value="${db.password}" />
</bean>
 
<!-- JPA TEMPLATE -->
<bean id="jpaTemplate" class="org.springframework.orm.jpa.JpaTemplate">
 <property name="entityManagerFactory" ref="emf-p" />
</bean>
 
<!-- DAOs -->
<bean id="entityOneDao" class="hes.db.impl.EntityOneDAOImpl">
  <property name="jpaTemplate" ref="jpaTemplate"/>
</bean>
 
<!-- PERSISTENCE UNIT -->
<bean id="pum" 
  class="org.springframework.orm.jpa.persistenceunit.DefaultPersistenceUnitManager">
  <property name="persistenceXmlLocations">
    <list>
      <value>META-INF/persistence.xml</value>
    </list>
  </property>
  <property name="dataSources">
    <map>
      <entry key="remoteDataSource" value-ref="dataSource-p" />
    </map>
  </property>
  <property name="defaultDataSource" ref="dataSource-p"/>
</bean>

Some of this may be redundant and will be cleaned up later.

If you have some better ideas, please do share. Thanks.

]]> http://dev.nirmalya.net/articles/java/spring-jpa-hibernate-tomcat-ehcache/feed 5 AnnotationException : Entity X References An Unknown Entity Y http://dev.nirmalya.net/articles/java/annotationexception-entity-x-references-an-unknown-entity-y http://dev.nirmalya.net/articles/java/annotationexception-entity-x-references-an-unknown-entity-y#comments Thu, 29 Apr 2010 18:33:43 +0000 nirmalya http://dev.nirmalya.net/articles/?p=296 Problem: I was getting an org.hibernate.AnnotationException when creating a bidirectional one-to-one relationship between 2 entities (EntityOne and EntityTwo).

Relevant Section Of StackTrace:
Caused by: org.hibernate.AnnotationException: @OneToOne or @ManyToOne on com.xyz.domain.EntityOne.anImage references an unknown entity: com.xyz.domain.EntityTwo
at org.hibernate.cfg.ToOneFkSecondPass.doSecondPass(ToOneFkSecondPass.java:103)
at org.hibernate.cfg.AnnotationConfiguration.processEndOfQueue(AnnotationConfiguration.java:541)
at org.hibernate.cfg.AnnotationConfiguration.processFkSecondPassInOrder(AnnotationConfiguration.java:523)
at org.hibernate.cfg.AnnotationConfiguration.secondPassCompile(AnnotationConfiguration.java:380)
at org.hibernate.cfg.Configuration.buildMappings(Configuration.java:1206)
at org.hibernate.ejb.Ejb3Configuration.buildMappings(Ejb3Configuration.java:1449)

Relevant Section Of EntityOne’s Code:

@Entity
@Table(name = "entity1")
public class EntityOne implements Serializable {
  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  @Column(name = "e1_id")
  private Long id;
 
  @OneToOne(fetch = FetchType.LAZY, optional = true)
  @JoinTable(name = "entity1_entity2_map",
    joinColumns = @JoinColumn(name = "e1_id"),
    inverseJoinColumns = @JoinColumn(name = "e2_id"))
  private EntityTwo anImage;
 
  //Other code
}

Relevant Section Of EntityTwo’s Code:

@Entity
@Table(name = "entity2")
public class EntityTwo implements Serializable {
  @Id
  @GeneratedValue(strategy = GenerationType.AUTO)
  @Column(name = "e2_id")
  private Long id;
 
  @OneToOne(mappedBy = "anImage", optional = true)
  private EntityOne e1;
 
  //Other code
}

Solution That Worked For Me:
Check the persistence.xml to ensure that the 2 entities (EntityOne and EntityTwo) are listed. Previously I had not included EntityTwo, which is why I kept getting the message com.xyz.domain.EntityOne.anImage references an unknown entity: com.xyz.domain.EntityTwo

    com.xyz.domain.EntityOne
    com.xyz.domain.EntityTwo
    <!-- other configurations -->
]]> http://dev.nirmalya.net/articles/java/annotationexception-entity-x-references-an-unknown-entity-y/feed 1