ROUGE or Recall-Oriented Understudy for Gisting Evaluation is a set of metrics and a software package used for evaluating automatic summarization and machine translation software in natural language processing.The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation.
While working on my Project on Summarisation, in order to compare my generated summary against the reference summary I had to set up ROUGE. In contrast to how there’s a lot of content on every topic available on the internet, well, installing ROUGE only has a few articles. None of them provided the sequence of steps and the requirements to install it. I wished there was some blog written. The content was present but I had to piece it all together from multiple sources to set it up. But, let me make things easy for the future researchers here in this post.
1. Download ROUGE-1.5.5 from here You only need the ROUGE-1.5.5 directory.
git clone https://github.com/andersjo/pyrouge.git cd pyrouge/tools/ROUGE-1.5.5
2. Check if Perl is installed else install it.
In Ubuntu, do
sudo apt-get install perl
3. For installing XML:DOM(this is a requirement for ROUGE to work) we install synaptic package manager
sudo apt-get update sudo apt-get install synaptic
4. Once Synaptic Package manager is installed, search for Synaptic package manager in your applications and launch it.
Once the package manager is opened search “libxml-dom-perl”
Click on Mark for Installation and apply changes.
5. An environment variable ROUGE_EVAL_HOME must be set to point to the data directory.
6. To avoid any WordNet exceptions, run these commands.
cd data/WordNet-2.0-Exceptions/ ./buildExeptionDB.pl . exc WordNet-2.0.exc.db cd ../ ln -s WordNet-2.0-Exceptions/WordNet-2.0.exc.db WordNet-2.0.exc.db
ROUGE is now installed, hurray!
Setting up pyrouge
pyrouge is a Python wrapper for the ROUGE summarization evaluation package. Getting ROUGE to work can require quite a bit of time. pyrouge is designed to make getting ROUGE scores easier by automatically converting your summaries into a format ROUGE understands, and automatically generating the ROUGE configuration file.
As of now, pypi version of pyrouge is deprecated, so let’s get the latest version from the repository
https://github.com/bheinzerling/pyrouge.git cd pyrouge
Set the ROUGE path with the command
Install pyrouge using
sudo python setup.py install
Test if everything’s installed by running
If the above command outputs “OK” , everything has been installed properly.