Saturday, August 22, 2020

How to install Moviepy on a AWS EC2

Recently, I wanted to move a python application from my local windows machine to the AWS EC2 instance. The application uses Moviepy for audio and video rendering. It took quite a bit of effort to port the application to AWS because the Moviepy intern uses FFMPEG and ImageMagick modules for rendering.

The below tutorial helps you install and configure the Moviepy module and its dependencies on an AWS EC2 instance. I assume you already have an EC2 instance up and running, refer to AWS documentation if you need help to launch an EC2 instance and SSH into the instance.

Table of content:

  1. Update the AWS EC2 instance and activate python 3
  2. Install FFMPEG
  3. Install and configure ImageMagick
  4. Install Moviepy

Step 1: Update the AWS EC2 instance and activate python 3

Execute below command to update the EC2 instance.

sudo amazon-linux-extras install python3 && sudo yum update

Validate whether python is activated using below command.

python3 -V

Step 2: Install FFMPEG

Refer to this Medium link to install FFMPEG on AWS EC2 instance. The end goal is to make sure the below command returns a response when you execute it.


Step 3: Install and configure ImageMagick

Execute the below command to install ImageMagick

sudo yum install ImageMagick

Update the policy.xml file and comment all the policies. (you will get unauthorized error if you miss this step)

sudo nano /etc/ImageMagick/policy.xml

Execute the below command to make sure ImageMagick is installed and working as expected


Step 4: Install Moviepy

Below command will install moviepy python module along with its dependencies.

sudo pip3 install moviepy

That’s it, Moviepy will auto detect FFMPEG and CONVERT modules and you should not face any issues.

Tuesday, December 4, 2018

Java Crash Course : Garbage Collection

In Languages like C, C++ it is the responsibility of the developer to allocate/deallocate the memory needed for a program using malloc and calloc commands.

In Java, it is done by Java virtual machine automatically using the Garbage collector. In general, any object on the heap which cannot be referenced from the stack is eligible for garbage collection.

Garbage Collection uses two strategies to clear the heap.

  1. Match and sweep - Instead of identifying the objects with no references, GC maps all the objects for which there is a reference from the stack and sweeps others.
  2. Generation Collection - It was discussed as part of Java Memory management crash course where the heap is divided into four sections. Objects from the young generation are swept more than the objects in the old generation.
Java provides two methods for interacting with Garbage collection.A developer has a minimum to no control on managing how and when to run garbage collection.

  1. System.gc() - Using which we can explicitly call garbage collection but there is no guarantee that it would execute. Further, while the garbage collector is running, all the threads are halted. i.e., the entire application is suspended.
  2. Finalize() - Object class has finalize method which can be overridden on your class as well to do some activity. Again, as in System.gc() there is no guarantee that this method is called when an object is garbage collected. The only valid use case is to see if the resources used are closed or not.

public void finalize()
log "object is discarded without releasing its resource"

If the maximum heap size allocated to the program is reached, the program will be terminated and java.lang.OutOfMemory error is thrown.

Java Crash Course : Memory Management in JVM

This document holds the summary of memory management concepts in JVM (Java Virtual Machine) that every software engineer should be aware of while designing/coding in Java.

Memory is divided into 2 parts
  1. Stack 
  2. Heap

Stack : Small when compared to heap memory and stores local variables and references to the objects in Heap.

  1. Each thread has its own Stack.
  2. Shorter life span
  3. Data from Stack is removed as soon as close braces ( } ) is encountered.
  4. Primitive types are saved in Stack.
  5. If a method pushed 10 elements to the Stack, all the 10 will be removed upon encountering closing braces.

Heap : This should be your main focus while designing/coding/debugging/testing the application.

  1. Long term memory.
  2. All objects are stored in the heap and references of those are saved on Stack.
  3. JDK provides jvisualvm application to analyse heap memory.
  4. In modern JVM heap is split into 2 types.
    1. Old Generation
    2. Young Generation - which in turn split into 3 types.
      • Eden
      • Survivor 0
      • Survivor 1
In older java versions 6 & 7, PermGen is also part of Heap memory which stores classes metadata and Strings which can never be garbage collected. From Java 8 PermGen/Metaspace is moved out of heap memory, and Strings are moved to Old Generation where they can be garbage collected.

Heap Memory to the Naked Eye

  • Above image shows the heap memory of a spring boot application where the maximum heap size set to 50mb.
  • For maven spring boot application.
    • set MAVEN_OPTS="-Xmx50m"
    • mvn spring-boot:run
  • Look how 50Mb is split among old, Eden, Survivor 1 and 2 memory.
    • Old Generation – 33.5 MB
    • Survivor 1 & 2 – 2.5 MB each
    • Eden – 15 MB
  • Objects are first allocated in Eden memory location if they survive the garbage collection, they are moved to survivor memory, then to Old generation. An object, in general, is moved to Old generation if it survives garbage collection more than eight times.
  • Garbage collector decides which memory should be cleared, i.e. Eden memory/ Old Gen memory based on the amount of memory it needs. In general, GC mostly clears the young memory and sometimes rarely old Gen.

Explore jvisualvm tool of JDK to monitor CPU, memory, threads, instances and get the heap dump.  Use eclipse memory analyser to see which class instances take up huge memory and which member variable is responsible for the memory leak.

Sunday, January 3, 2016

Jasper Localization From Property File:Encoding Issue

Recently In a Project I used Jasper to generate PDF forms.While doing this task I encountered a problem with Localization of Jasper Fields from Property Files.

Normally we include below attribute in JRXML header to point to localization files.

Localization files will be in same folder where JRXML is placed and localization files will be in below format.
  • etc
And each file will have key value pair like Date Date

and it will be used in JRXML in below format

and Localization is passed to JRXML in below format
 parameterMap.put(JRParameter.REPORT_LOCALE, new Locale("th"));
Encoding Issue:

When I generated the PDF for Thai language the output was a bunch of junk values.

The problem is resource bundle reads the property files in ASCII encoding even though property file is encoded in UTF-8


Remove the resourcebundle attribute from JRXML and configure it in Java class with UTF-8 Encoding.

  InputStream input = this.getClass().getClassLoader().getResourceAsStream("jasperreport_"+locale.getLanguage()+ ".properties");
  Reader reader = new InputStreamReader(input, "UTF-8");
  ResourceBundle resourceBundle = new PropertyResourceBundle(reader);
 catch (IOException  e1) 
  LOG.error("Error locating localize file ",e1);

Sunday, September 27, 2015

Your own Broken Link Checker

Python Script : Crawl a website ,take screenshots and save it in word doc.

Steps to execute this script
  1. Create a folder 'link Checker' and place this script in it.
  2. Create a folder 'shots' and place it inside  'link Checker' folder.
  3. Change below variables 
    • url ="website url"
    • SaveDirectory=r ''screenshot directory"
  4. Execute the program.

 import requests  
 from BeautifulSoup import BeautifulSoup  
 import webbrowser  
 import os  
 import sys  
 import time  
 import Image  
 import ImageGrab  
 import win32com.client as win32  
 import os  
 import time  
 import glob  
 url = ""  
 response = requests.get(url)  
 # parse html  
 page = str(BeautifulSoup(response.content))  
 alllist = [];  
 httplist = [];  
 otherlist = [];  
 SaveDirectory=r'C:\Users\Giridhar\Desktop\link Checker\shots'  
 def getURL(page):  
   :param page: html of web page (here: Python home page)   
   :return: urls in that page   
   start_link = page.find("a href")  
   if start_link == -1:  
     return None, 0  
   start_quote = page.find('"', start_link)  
   end_quote = page.find('"', start_quote + 1)  
   url = page[start_quote + 1: end_quote]  
   return url, end_quote  
 while True:  
   url, n = getURL(page)  
   page = page[n:]  
   if url:  
     #print url  
 #print alllist  
 for httpurl in alllist:    
   if httpurl.find("http")!=-1:  
     if httpurl.endswith("/",0,1):  
 #print httplist  
 #print otherlist  
 new = 0  
 for browsing in httplist:  
   print browsing  
   i += 1  
   if i == 10:  
 #---------------------------------code to move images to word-------  
 allpics = []  
 wordApp = win32.gencache.EnsureDispatch('Word.Application') #create a word application object  
 wordApp.Visible = False # hide the word application  
 doc = wordApp.Documents.Add() # create a new application  
 for pic in allpics:  
   current_pic = doc.InlineShapes.AddPicture(pic)  
   current_pic.Height= 400  
   current_pic.Width= 400  

After execution of script ,it will automatically open the default browser and crawl all the pages it in and takes screenshot of all pages.

Once done with the screenshots it will create a doc file with all screenshots in it.

Wednesday, February 18, 2015

What is the quickest traffic strategy to get traffic to your blog?

Quickest strategies will only work for a short period (A day or two).
Below strategy will help you to get huge traffic in very short duration and for a very short time.
Mantra: Target on NARROW LOCAL HOT topics.

Eg: Keep an eye on local exam results like GATE/IIT/JNTU/10th etc
I will explain taking JNTU as an example. You can apply same concept to other topics as well
 Do you know how many hits a JNTU website gets during semester results?

According to Alexa, JNTU website will get 40K hits during semester results. 
 Do you know how users reach JNTU results webpage?

Yup they Google it.
Now here is the tricky part .What if Google shows your blog as the first result when user search for "jntu results". YOU WILL GET 40K visits in a single day.

Friday, October 24, 2014

Control Remote Car with Keyboard

I got bored with software projects and thought of doing something different which I never tried. I've  always been fascinated by robots and have a dream of building one.This small baby step project helped me to understand the various concepts of Electronics.