Hepatocellular carcinoma (HCC) is one of the most critical health problems in the world. For proper treatment, it is important to identify the grade of cancer morbidity from HCC biopsy image. The diagnostic work is not only time-consuming but also subjective. The same biopsy image may be diagnosed as of different grades by different doctors, due to lack of experience or difference in opinion. In this work, we proposed an automatic grading system with classification accuracy matching to an experienced doctor, to help augment the diagnosis process. First, we proposed a segmentation method to isolate all nucleus-like objects present in a biopsy image. Non-target objects (here the target is a single HCC nucleus) present in the biopsy image are isolated too in the segmentation process. To eliminate such non-target objects, we proposed clustering of segmented images and a novel method to filter out target objects. Next, we proposed a two track neural network, where input consists of 2 different images. It combines a single segmented nucleus and a random cropped texture patch of the biopsy image to which the nucleus belongs. At this classifier output, we grade the single nucleus. Finally, a majority voting method is used to identify the grade of the whole biopsy image. We achieved an accuracy of 99.03% for nucleus image grading and 99.66% accuracy for grading biopsy images.