PJT FINAL.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5cc695b2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pypdf2\n",
      "  Obtaining dependency information for pypdf2 from https://files.pythonhosted.org/packages/8e/5e/c86a5643653825d3c913719e788e41386bee415c2b87b4f955432f2de6b2/pypdf2-3.0.1-py3-none-any.whl.metadata\n",
      "  Downloading pypdf2-3.0.1-py3-none-any.whl.metadata (6.8 kB)\n",
      "Downloading pypdf2-3.0.1-py3-none-any.whl (232 kB)\n",
      "   ---------------------------------------- 0.0/232.6 kB ? eta -:--:--\n",
      "   ----- --------------------------------- 30.7/232.6 kB 640.0 kB/s eta 0:00:01\n",
      "   ------ -------------------------------- 41.0/232.6 kB 487.6 kB/s eta 0:00:01\n",
      "   ---------- ---------------------------- 61.4/232.6 kB 544.7 kB/s eta 0:00:01\n",
      "   --------------- ----------------------- 92.2/232.6 kB 476.3 kB/s eta 0:00:01\n",
      "   ------------------ ------------------- 112.6/232.6 kB 502.0 kB/s eta 0:00:01\n",
      "   --------------------- ---------------- 133.1/232.6 kB 462.0 kB/s eta 0:00:01\n",
      "   -------------------------- ----------- 163.8/232.6 kB 490.7 kB/s eta 0:00:01\n",
      "   ---------------------------- --------- 174.1/232.6 kB 476.3 kB/s eta 0:00:01\n",
      "   --------------------------------- ---- 204.8/232.6 kB 478.0 kB/s eta 0:00:01\n",
      "   ------------------------------------ - 225.3/232.6 kB 491.0 kB/s eta 0:00:01\n",
      "   -------------------------------------- 232.6/232.6 kB 474.1 kB/s eta 0:00:00\n",
      "Installing collected packages: pypdf2\n",
      "Successfully installed pypdf2-3.0.1\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install pypdf2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3f8bbd22",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Pages: 10\n",
      "Text from all pages:\n",
      "Image De -Hazing based on Krill Herd \n",
      "Optimization algorithm  \n",
      "xxx1, *yyy1, zzz3 \n",
      "email slots  \n",
      "1. xxx,  \n",
      "2. yyy   \n",
      "3. zzz \n",
      " \n",
      "Abstract :  \n",
      " In digital images, haze damages the image quality which are taken in outdoor environment. \n",
      "This causes the degradation of image quality in terms of color distortion and there exists a huge  loss in \n",
      "the contrast.  Because of the difficulty and importance of the problem, a lot of study has been done on \n",
      "picture de -hazing.  A significant number of approaches were done based on  dark channel prior shows  \n",
      "successful  results  among picture haze removal approaches.  Furthermore, the addition of a guided filter \n",
      "has greatly improved its efficiency.  To address this issue, a n optimization approach -based  Image -\n",
      "Dehazing technique is applied in this research paper. It generates pixel -wised transmission without \n",
      "additional refining.  Furthermore, image de -hazing procedures based on noise filtering perspective  are \n",
      "used to achieve the goal of saturation enhancement while adhering to the minimum hue change \n",
      "constraint.  The usefulness and efficiency of the suggested algorithm were evaluated by qualitative and \n",
      "quantitative analysis of the results.  \n",
      "Keywords:  De-Hazing, Krill Herd Optimization algorithm, Noise filtering.  \n",
      "I. Introduction  \n",
      "When taking images outside, a large volume of air particles will damage the image quality.  Haze, \n",
      "fog, and smoke are all categorized  as haze because they have a similar effect on visual readability.  \n",
      "Furthermore, color  distortion is caused by air interference [1].  In a range of applications, clear, high -\n",
      "quality images with high saturation, contrast, entropy, and other quality criteria are crucial [2,3].  \n",
      "This is a difficult endeavour, and much study has gone into tackling the pollution issue.  In recent \n",
      "years, various methods for image de -hazing have been presented; Liu [4] published a study and \n",
      "categorization in 2015, on Dark Channel Prior (DCP) [5].  Multiple image de -hazing [2,6,7], haze \n",
      "removal requiring additional information [8 –10], and single image de -hazing [11 –12, 25] are the three \n",
      "types of image haze removal methods.  Due to the additional  requirement of  resources high  \n",
      "computational  complexity, the first two techniques are not suitable for real -time applications.  Single \n",
      "image de -hazing has grabbed the interest of many scholars due to its simplicity and efficacy in picture \n",
      "de-hazing.  He [11] proposed a technique based on the notion of DCP in 2011, and it is by far the mos t \n",
      "impressive single picture de -hazing solution to date.  \n",
      "This paper describes an approach known as image Krill Herd - Haze ( KH-H), which is based on a \n",
      "notion proposed by Gandomi, et al.  [13]. High intensity and low saturation are the main characteristics \n",
      "of pictures polluted by haze in the form of noise . As a consequence, the intensity and saturation of the \n",
      "input picture are weighted to reflect the hazy severity  [24]. \n",
      "The same approach has been utilized  to estimate light  ambience ; however, when images contain \n",
      "excessively bright objects, a minor modification is necessary.  Picture noise filtering uses local data from \n",
      "the severity map after the two weighted maps have been constructed.  Four parameters are optimized \n",
      "using the Krill Herd method.  The purpose of this project is to increase the saturation of the final image.  \n",
      "The further paper is organized as follows: Section 2 discusses on the related work carried out on \n",
      "de-hazing the images. Section 3 discusses on image dehazing problem definition. Section 4 discusses, \n",
      "on the Krill Herd Algorithm  and its application to solve image De -Hazing. Section 5 discusses on the \n",
      "experimental results and its analysis and finally section 6 concludes the paper . \n",
      "II. Related Works  \n",
      "The RGB -HSV conversion [ 14] is used to turn the colour image into its monochrome counterpart.  \n",
      "The intensity (V -channel) is treated for contrast enhancement before being displayed in RGB format  \n",
      "(CHE). The Mean Brightness Preserving Bi -Histogram Equalization (BBHE) described in [ 15] was \n",
      "created for contrast augmentation while maintaining the output image's mean brightness.  In concept, \n",
      "the Dualistic Sub -Image Histogram Equalization (DSIHE) [ 16] is comparable to BBHE.  The sole \n",
      "difference is that DSIHE divides the input picture intensity into two sub -images using the median value \n",
      "rather than the mean value.  The median value is based on the picture content and may be less than or \n",
      "bigger than the mean value.  The technique of Recursive Mean -Separate Histogram Equalization \n",
      "(RMSHE) was proposed in [ 17]. This approach may be thought of as a more advanced variant of BBHE.  \n",
      "The split into sub -images, in instance, is performed four times to generate four sub -images.  \n",
      "Recursive Sub -Picture Histogram Equalisation (RSIHE) [ 18] took up the notion of dividing the \n",
      "input image into four components.  The division was changed to be based on the deciles of the number \n",
      "of pixels in the ordered ascending intensity values.  The rest of the enhancing technique follows the \n",
      "same steps as the previous procedures.  The Fuzzy Fusion based High Dynamic Range Imaging using \n",
      "Adaptive Histogram Separation (FFHAHS) method, which divides the picture according to an intensity \n",
      "threshold, was disclosed  in [19]. Range stretching and range clipped histogram equalisation are done \n",
      "once two sub -images are created.  The final improved picture is created by fusing the input and two \n",
      "enhanced intermediate images using fuzzy inference.  \n",
      "III. Image Dehazing  \n",
      "The notion of picture dehazing is to recover the real scene radiance of an image from ambient light \n",
      "(A) and the distance from the object to the camera lens  (t) [23]. A pixel in a picture may be described \n",
      "formally as  \n",
      "𝑃(𝑖)=𝐽(𝑖)×𝑡(𝑖)+𝐴(1−𝑡(𝑖)) \n",
      "where P denotes the real intensity as RGB values, J is the  radiance value at distance t, and A denotes \n",
      "ambient light.  However, for coloured images, the image is a collection of pixels from a single layer (in \n",
      "the case of GrayScale images) or three layers (RGB).  As a result, the key goal is to estimate the ambient \n",
      "light intensity and the distance between the camera and the object in order to extract radiance from pixel \n",
      "values.  \n",
      "According to a recent study, scene radiance may be recovered from collected pixels using noise filtering . \n",
      "The standard image representation has been recast in this style as follows:  \n",
      "𝑃(𝑖)−𝐴=(𝐽(𝑖)−𝐴)×𝑛(𝑖) \n",
      " where the transmission distance ( 𝑡) is treated as noise disturbance ( 𝑛). \n",
      "According to the investigations, the pixel value influenced by haze has a high brightness value (B) \n",
      "but low saturation (S).  The noise on a picture may be reframmed as follows based on these parameters \n",
      "B and S:  \n",
      "𝑀𝑛=𝜌(1−𝐵)+(1−𝜌)𝑆 \n",
      "where B is the average of the image's R ed, Green, and B lue (RGB)  colours, and S is the difference \n",
      "between the numeric value 1 and the pixel's minimum value P to the maximum value pixel's ratio.  \n",
      "Based on the scenic radiance, the parameter indicates the weight factor that limits the participation of \n",
      "brightness value and saturation.  \n",
      " \n",
      "where 𝐵 is the average of R, G, B colors of image and S is the difference between 𝑃 (minimum value \n",
      "of a pixel)  to the ratio of maximum value pixel. The parameter 𝜌 represents the weight factor that \n",
      "restricts the involvement of brightness value and saturation based on the scenic radiance  [22].  \n",
      " The noise map has been fine -tuned based on the factor  of shift -scale  as follows:  \n",
      "𝑀𝑠=𝛼+𝛽𝑀𝑛 \n",
      "Now that the total noise has been calculated, the final pixel creation in the real picture (i.e. Eq.(2)) \n",
      "may be reorganised as follows:  \n",
      "𝑃(𝑖)−𝐴=(𝐽(𝑖)−𝐴)×𝑀𝑠(𝑖) \n",
      "Apart from noise caused by transmission distance, ambient light has an impact on scenic brilliance.  And \n",
      "because this is linked to the brightness and saturation value, another constricting component, 𝑀𝑎, is \n",
      "generated from B and S as follows : \n",
      "𝑀𝑎=𝜑𝐵+(1−𝜑)𝑆 \n",
      "where 𝜑 is the weight factor. And thus the value of 𝐴 can be identified as  \n",
      "𝐴=max {𝑀𝑎(𝑖)} \n",
      "The final A can be cummulated as  \n",
      "𝐴=𝐴+𝛾 \n",
      "where 𝛾 is a correlation scalar value related to the atmospheric light.  \n",
      "When dealing with all of the values of  𝜌,𝛼,𝛽,𝛾 as weight factors, it should be ideal such that they do \n",
      "not deviate from the scene's real radiance value.  As a result, the parameter values are optimised using a \n",
      "Krill Herd  optimization model.   \n",
      "IV. Krill -Herd  based Image Dehazing  \n",
      "Bio-inspired algorithms are important in the field of optimization, where a better solution must be \n",
      "found for a situation that has an exponential number of viable answers proportionate to the magnitude \n",
      "of the issue.  Krill Herd algorithm  [13] (KH) is an effective optimization technique that is used to address \n",
      "a large number of issues in bioinspired optimization algorithms.  KH's working style:  When predators \n",
      "attack krill, like as seals, penguins, or seagulls, they take individual krill with them.  As a result,  the krill \n",
      "density decreases  [20].  \n",
      "Many factors influence the establishment of the krill herd following predation.  Individual krill \n",
      "herding is a multi -objective process with two primary goals: (1) to increase density  of the krill  and (2) \n",
      "obtaining the food.  This technique is used to propose a swarm based  algorithm for tackling global \n",
      "optimization issues in the current study.  The krill's density -dependent attraction (growing density) and \n",
      "the search for food are employed as goals, leading to the krill herding around the global minima  [21]. \n",
      "When a krill seeks for the maximum density and food in this process, it travels tow ard the optimal \n",
      "option.  That is, the goal function decreases as the distance between the high density and food increases.  \n",
      "The weight factors of  𝜌,𝛼,𝛽,𝛾 will be used as input in this Krill -Herd optimization technique to \n",
      "obtain the optimum values.  The aim function that must be maximised is the output picture saturation.  \n",
      "Increased picture saturation without bounds, on the other hand, will result in an artificial output that \n",
      "fails to convey the true information present in the original image.  As a result, particles that have seen a \n",
      "considerable colour shift as a result of the dehazing method are excluded using a penalty factor ( ℵ). \n",
      "ℵ=1−ℰ \n",
      "where ℰ is the value of 𝑡-test for every individual in the population. Every individual's fitness value \n",
      "may be calculated using the fitness function, which is theoretically defined as follows:  \n",
      "𝑓(𝑥𝑖)=(1−ℵ)𝑆̂+ℵ(1−ℵ) \n",
      "where 𝑆̂ is the average saturation rate. Algorithm1 contains the algorithm for determining optimal \n",
      "parameter values . \n",
      "  \n",
      "Image De -Hazing using Krill Herd Optimization algorithm  \n",
      "Input:  The parameters 𝜌,𝛼,𝛽,𝛾 upper and lower bound , Objective function (f)  \n",
      " \n",
      "Step 1 : Initialize  𝐼𝑡𝑒𝑟 ←1,𝑖←0,𝑘←0 \n",
      "Step 2:  ∀ 𝐼𝑛𝑑𝐾𝐻 ∈𝑁𝐾𝐻 do  \n",
      "      𝑃𝐼𝑛𝑑𝐾𝐻,𝑗←𝐿𝐵𝑗+(𝑈𝐵𝑗−𝑈𝐵𝑗)∗𝑟𝑎𝑛𝑑 ()  | 𝑗∈𝜌,𝛼,𝛽,𝛾  \n",
      " end for  \n",
      "Step 3 : ∀ 𝐼𝑛𝑑𝐾𝐻 ∈𝑁𝐾𝐻  do   \n",
      "      𝐹𝑖𝑡(𝑃𝐼𝑛𝑑𝐾𝐻)←𝑓(𝑃𝐼𝑛𝑑𝐾𝐻)     \n",
      " end for  \n",
      " \n",
      "Repeat  \n",
      " \n",
      "Step 4: Movement Induced by other Krills  \n",
      "Step 4.1: Repeat till 4 .3 Until  𝑖< 𝑁𝐾𝐻  | 𝑖∈𝐼𝑛𝑑𝐾𝐻 else Step 5 \n",
      "       Step 4.2: 𝜗𝑖= 𝜗𝑖𝑙𝑜𝑐𝑎𝑙+ 𝜗𝑖𝑡𝑎𝑟𝑔𝑒𝑡     \n",
      "       Step 4.3: 𝑁𝑖𝑛𝑒𝑤= 𝑁𝑚𝑎𝑥 𝜗𝑖+ 𝜔𝑛 𝑁𝑖𝑜𝑙𝑑      \n",
      " \n",
      "Step 5: Foraging motion of individual krill \n",
      "Step 5.1: Repeat till Step 5.3Until  𝑖< 𝑁𝐾𝐻  | 𝑖∈𝐼𝑛𝑑𝐾𝐻  else Step 6 \n",
      "       Step 5.2: 𝜑𝑖=𝜑𝑖𝑓𝑜𝑜𝑑+𝜑𝑖𝑏𝑒𝑠𝑡      \n",
      "       Step 5.3: 𝐹𝑖= 𝑉𝑓𝛽𝑖+𝜔𝑓𝐹𝑖𝑜𝑙𝑑  \n",
      " \n",
      "Step 6: Individual movement of Krill Herd in random manner  \n",
      "Step 6.1: Repeat till Step 6.2Until   𝑖<𝑁𝐾𝐻 | 𝑖∈𝐼𝑛𝑑𝐾𝐻  else Step 7 \n",
      "       Step 6.2:  𝐷𝑖= 𝐷𝑚𝑎𝑥 (1−𝐼\n",
      "𝐼𝑚𝑎𝑥) \n",
      " \n",
      "Step 7: Repeat till Step 7.2 Until   𝑖<𝑁𝐾𝐻 | 𝑖∈𝐼𝑛𝑑𝐾𝐻 else Step 8 \n",
      "       Step 8.1: 𝑑𝑋𝑖\n",
      "𝑑𝑡= 𝑁𝑖+ 𝐹𝑖+𝐷𝑖       \n",
      "       Step 8.2:   ∆𝑡= ∁𝑡 ∑ (𝑈𝐵𝑗− 𝐿𝐵𝑗)𝑁𝑉\n",
      "𝑗=1      \n",
      "       Step 8.3:   𝑋𝑖(𝑡+∆𝑡)= 𝑋𝑖(𝑡)+∆𝑡 𝑑𝑋𝑖\n",
      "𝑑𝑡       \n",
      " \n",
      "Until 𝑀𝑎𝑥𝐼𝑇≤𝐼𝑡𝑒𝑟  \n",
      " \n",
      "Step 9 : Return min  (𝐹𝑖𝑡(𝑃𝐼𝑛𝑑𝐾𝐻)) \n",
      " \n",
      "Output: min  (𝐹𝑖𝑡(𝑃𝐼𝑛𝑑𝐾𝐻)) \n",
      " \n",
      " \n",
      "Figure 5: Flow Chart  of Krill Herd Algorithm  for Ima ge De -Hazing  \n",
      "  \n",
      "\n",
      "V. Experimental Analysis  \n",
      "To test the effectiveness and applicability of the presented approach, an experiment was conducted \n",
      "on 100 colour images taken under various environmental conditions.  They include images with haze, \n",
      "poor lighting, and low contrast.  These pictures are saved in the 8 -bit JPEG format.  The computer scripts  \n",
      "were written in MATLAB  2018a and run on a PC with a Core i7 1 0th Gen CPU, 16GB RAM, and a \n",
      "256GB  SD. \n",
      "Table 1: Sample results of KH-H \n",
      "Si.\n",
      "No Haze affected Image  De-hazed image using KH-H \n",
      "1 \n",
      "  \n",
      "2 \n",
      "  \n",
      "3 \n",
      "  \n",
      "\n",
      "4 \n",
      "  \n",
      " \n",
      "In Table 2, the performance metrices  values  of KH-H such as colourfulness , saturation, contrast, \n",
      "sharpness, mean brightness,  and entropy  are all compared with the other existing algorithms  \n",
      " \n",
      " \n",
      " \n",
      "Table 2: Comparison of Mean values of KH-H vs other existing algorithms  \n",
      " Colorfulness  saturation  contrast  Sharpness  Mean -\n",
      "brightness  Entropy  \n",
      "Fattal 08  0.099  0.316  0.023  0.020  0.363  6.267  \n",
      "Tarel 09  0.167  0.364  0.046  0.048  0.397  6.927  \n",
      "Tarel 10  0.153  0.502  0.032  0.041  0.226  6.618  \n",
      "He11  0.163  0.539  0.031  0.024  0.205  6.425  \n",
      "Meng13  0.140  0.425  0.042  0.036  0.250  6.706  \n",
      "Zhu 15 0.125  0.286  0.030  0.030  0.370  6.959  \n",
      "KH-H 0.110 0.542 0.041 0.038 0.221 6.221  \n",
      " \n",
      "On comparing the results of existing approaches, KH-H outperforms the existing approaches in a \n",
      "varied number of metrices.  \n",
      "In Colorfulness, KH -H outperforms all the existing approaches except Fattal 08 which goes less \n",
      "than 0.1. In terms of saturation KH -H outperforms all the existing algorithms and the second level of \n",
      "algorithm goes with Hel 1 which goes less with 0.003 points  in terms of saturation. In the contrast model \n",
      "KH-H outperforms all the existing  approaches except Hel 1 with just 0.001 unit of contrast value.  \n",
      "On comparing the value for sharpness, the proposed model outperforms with half of the existing \n",
      "approaches  and at the same time it performs less when compared with Tarel 09 and Tarel 10. The mean \n",
      "brightness is optimal when compared with all other existing approaches and the Entropy value seems \n",
      "to be very compromising when compared with existing models .  \n",
      " \n",
      "VI. Conclusion  \n",
      "This research provides a novel method for de -hazing photos that use a noise filtering approach to \n",
      "restore haze -free images.  In regulating the depth shift near image edges, the quantitative investigation \n",
      "\n",
      "reveals that pixel -wise noise estimate is more precise than local -patch based solutions.  Another benefit \n",
      "of the suggested KH -H method is that it is unaffected by images with extremely bright objects.  This is \n",
      "performed by making a correction to the atmospheric value's estimated value.  The versatility of the \n",
      "recommended technique is further extended using  the Krill Herd Algorithm to automate parameter \n",
      "changes for each input picture.  The purpose of this optimization technique with a hue change constraint \n",
      "is to maximize saturation in the final image.  \n",
      "References  \n",
      "[1] Kim J. -G.. Color correction device for correcting color distortion and gamma characteristic. 1999. US Patent 5,949,496  \n",
      "[2] Narasimhan SG, Nayar SK. Contrast restoration of weather degraded images. Pattern Anal Mach Intell IEEE Trans \n",
      "2003a;25(6):713 –24. \n",
      "[3] Henry RC, Mahadev S, Urquijo S, Chitwood D. Color perception through atmospheric haze. JOSA A 2000;17(5):831 –5. \n",
      "[4] Liu S, Rahman M, Wong C, Lin S, Jiang G, Kwok N. Dark channel prior based image de -hazing: a review. In: Information science \n",
      "and technology (ICIST), 2015 5th international conference on. IEEE; 2015. p. 345 –50. \n",
      "[5] Lee S, Yun S, Nam J -H, Won CS, Jung S -W. A review on dark channel prior based image dehazing algorithms. EURASIP J Image \n",
      "Video Process  2016;2016(1):1 –23. \n",
      "[6] Schechner YY, Narasimhan SG, Nayar SK. Polarization -based vision through haze. Appl Opt 2003;42(3):511 –25 \n",
      "[7] Nayar SK, Narasimhan SG. Vision in bad weather. In: Computer vision, 1999. The proceedings of the seventh IEEE international \n",
      "conference on, 2. IEEE; 1999. p. 820 –7. \n",
      "[8] Tan K, Oakley J. Enhancement of color images in poor visibility conditions. In: Image processing, 2000. Proceedings. 2000 \n",
      "international conference on, 2. IEEE; 2000. p. 788 –91. \n",
      "[9] Narasimhan SG, Nayar SK. Interactive (de) weathering of an image using physical models. In: IEEE workshop on color and \n",
      "photometric methods in computer vision, 6. France; 2003b. p. 1.  \n",
      "[10] Kopf J, Neubert B, Chen B, Cohen M, Cohen -Or D, Deussen O, et al. Deep photo: model -based photograph enhancement and  \n",
      "viewing. In: ACM transactions on graphics (TOG), 27. ACM; 2008. p. 116.  \n",
      "[11] He K, Sun J, Tang X. Single image haze removal using dark channel prior. Pattern Anal Mach Intell IEEE Trans 2011;33(12):2341  \n",
      "53. \n",
      "[12] Lee J -S. Digital image enhancement and noise filtering by use of local statistics. Pattern Anal Mach Intell IEEE Trans 1980(2):165 –\n",
      "8 \n",
      "[13] Gandomi, Amir Hossein, and Amir Hossein Alavi. \"Krill herd: a new bio -inspired optimization algorithm.\" Communications in \n",
      "nonlinear science and numerical simulation 17.12 (2012): 4831 -4845.  \n",
      "[14] R. C. Gonzalez and R. E. Woods, Digital Image Processing, 3rd ed. Prentice -Hall Inc., Upper Saddle River, NJ, USA, 2006  \n",
      "[15] Y.-T. Kim, “Contrast enhancement using brightness preserving bi -histogram equalization,” Consumer Electronics, IEEE  \n",
      "Transactions on, vol. 43, no. 1, pp. 1 –8, 1997  \n",
      "[16] Y. Wang, Q. Chen, and B. Zhang, “Image enhancement based on equal area dualistic sub -image histogram equalization method,” \n",
      "Consumer Electronics, IEEE  Transactions on, vol. 45, no. 1, pp. 68 –75, 1999.  \n",
      "[17] S.-D. Chen and A. R. Ramli, “Contrast enhancement using recursive mean -separate  histogram equalization for scalable brightness \n",
      "preservation,” Consumer Electronics, IEEE Transactions on, vol. 49, no. 4, pp. 1301 –1309, 2003.  \n",
      "[18] K. Sim, C. Tso, and Y. Tan, “Recursive sub -image histogram equalization applied  to gray scale images,” Pattern Recognition \n",
      "Letters, vol. 28, no. 10, pp. 1209 –1221,  2007.  \n",
      "[19] R. Duvar, O. Urhan et al., “Fuzzy fusion based high dynamic range imaging using adaptive histogram separation,” IEEE \n",
      "Transactions on Consumer Electronics,  vol. 61, no. 1, pp. 119 –127, 2015.  \n",
      "[20] Darong Huang, Zhou Fang, Ling Zhao, Xiaoyan Chu, \"An improved image clearness algorithm based on dark channel prior\", \n",
      "Proceedings of the 33rd Chinese Control Conference, pp.7350 -7355, 2014.  \n",
      "[21] Wei-Jheng Wang, Bo -Hao Chen, Shih -Chia Huang, \"A Novel Visibility Restoration Algorithm for Single Hazy Images\", 2013 \n",
      "IEEE International Conference on Systems, Man, and Cybernetics, pp.847 -851, 2013.  \n",
      "[22] Wilawun Punmanee, Teerasit Kasetkasem, Thitiporn Chanwimaluang, Akinori Nishihara, \"Naturalness color enhancement for \n",
      "THEOS images\", 2013 13th International Symposium on Communications and Information Technologies (ISCIT), pp.523 -528, \n",
      "2013.  \n",
      "[23] Xue-Hui Wei, Meng Zhao, Zhi -Hai Sun, \"Analysis of color channels effection on image compression distortion\", 2012 \n",
      "International Conference on Wavelet Active Media Technology and Information Processing (ICWAMTIP), pp.157 -160, 2012  \n",
      "[24] Yishu Zhai, Yong Zhang, \"Contrast Restoration for Fog -Degraded Images\", 2009 International Conference on Computational \n",
      "Intelligence and Security, vol.1, pp.619 -623, 2009.  \n",
      "[25] Kaiqi Huang, Qiao Wang, Zhenyang Wu, \"Color image enhancement and evaluation algorithm based on human visual system\", \n",
      "2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol.3, pp.iii -721, 2004.  \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      "Enhancement of color images in poor \n",
      "visibility conditions  \n",
      "Abstract:  \n",
      "Degradation of images by the atmosphere often restricts imaging applications to good \n",
      "visibility conditions. For example, when imaging the terrain from a forward -looking airborne \n",
      "camera, the atmospheric degradation causes both a loss in contrast and color information. \n",
      "Enhancement of such images is a difficult ta sk due to the complexity in restoring both the \n",
      "luminance and chrominance while maintaining good color fidelity. One particular problem is \n",
      "the fact that the level of contrast loss depends strongly on the wavelength; shorter \n",
      "wavelengths i.e., blue are more e ffected. In this paper, a novel method is presented for the \n",
      "enhancement of color images. This method is based on the underlying physics of the \n",
      "degradation and the parameters required for enhancement are estimated from the image itself. \n",
      "The proposed method is tested using synthetic images to explore the limitations and \n",
      "reliability of the method under different visibility conditions. Enhancement is performed on \n",
      "real images taken using an airborne camera at a height of approximately 1000 meters in hazy \n",
      "conditi ons for which the visibility is approximately 10 kilometers. Significant improvements \n",
      "in terms of contrast, visible range and color fidelity are evident when compared to existing \n",
      "methods.  \n",
      " \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Import library\n",
    "import PyPDF2\n",
    "\n",
    "# Create an object for reading a file. Use the file path directly.\n",
    "pdfReader = PyPDF2.PdfReader(\"C:\\\\Users\\\\megha\\\\document.pdf\")\n",
    "\n",
    "# Get the number of pages in the pdf file using the recommended approach.\n",
    "numPages = len(pdfReader.pages)\n",
    "\n",
    "# Initialize a variable to hold all the text\n",
    "all_text = \"\"\n",
    "\n",
    "# Loop through all the pages\n",
    "for i in range(numPages):\n",
    "    # Get the current page\n",
    "    page = pdfReader.pages[i]\n",
    "    \n",
    "    # Extract text from the current page and add it to the all_text variable\n",
    "    text = page.extract_text()\n",
    "    all_text += text + \"\\n\"  # Add a newline character after each page's text for better readability\n",
    "\n",
    "# Print the number of pages and all text data.\n",
    "print(\"Number of Pages:\", numPages)\n",
    "print(\"Text from all pages:\\n\" + all_text)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4eb067dd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: nltk in c:\\users\\megha\\anaconda3\\anaconda\\lib\\site-packages (3.8.1)\n",
      "Requirement already satisfied: click in c:\\users\\megha\\anaconda3\\anaconda\\lib\\site-packages (from nltk) (8.0.4)\n",
      "Requirement already satisfied: joblib in c:\\users\\megha\\anaconda3\\anaconda\\lib\\site-packages (from nltk) (1.2.0)\n",
      "Requirement already satisfied: regex>=2021.8.3 in c:\\users\\megha\\anaconda3\\anaconda\\lib\\site-packages (from nltk) (2022.7.9)\n",
      "Requirement already satisfied: tqdm in c:\\users\\megha\\anaconda3\\anaconda\\lib\\site-packages (from nltk) (4.65.0)\n",
      "Requirement already satisfied: colorama in c:\\users\\megha\\anaconda3\\anaconda\\lib\\site-packages (from click->nltk) (0.4.6)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\megha\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Install NLTK (if not already installed)\n",
    "!pip install nltk\n",
    "\n",
    "# Download Punkt Tokenizer models\n",
    "import nltk\n",
    "nltk.download('punkt')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d9265b43",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Furthermore, color  distortion is caused by air interference [1].\n",
      "In a range of applications, clear, high -\n",
      "quality images with high saturation, contrast, entropy, and other quality criteria are crucial [2,3].\n",
      "In recent \n",
      "years, various methods for image de -hazing have been presented; Liu [4] published a study and \n",
      "categorization in 2015, on Dark Channel Prior (DCP) [5].\n",
      "Multiple image de -hazing [2,6,7], haze \n",
      "removal requiring additional information [8 –10], and single image de -hazing [11 –12, 25] are the three \n",
      "types of image haze removal methods.\n",
      "He [11] proposed a technique based on the notion of DCP in 2011, and it is by far the mos t \n",
      "impressive single picture de -hazing solution to date.\n",
      "[13].\n",
      "As a consequence, the intensity and saturation of the \n",
      "input picture are weighted to reflect the hazy severity  [24].\n",
      "Related Works  \n",
      "The RGB -HSV conversion [ 14] is used to turn the colour image into its monochrome counterpart.\n",
      "The Mean Brightness Preserving Bi -Histogram Equalization (BBHE) described in [ 15] was \n",
      "created for contrast augmentation while maintaining the output image's mean brightness.\n",
      "In concept, \n",
      "the Dualistic Sub -Image Histogram Equalization (DSIHE) [ 16] is comparable to BBHE.\n",
      "The technique of Recursive Mean -Separate Histogram Equalization \n",
      "(RMSHE) was proposed in [ 17].\n",
      "Recursive Sub -Picture Histogram Equalisation (RSIHE) [ 18] took up the notion of dividing the \n",
      "input image into four components.\n",
      "The Fuzzy Fusion based High Dynamic Range Imaging using \n",
      "Adaptive Histogram Separation (FFHAHS) method, which divides the picture according to an intensity \n",
      "threshold, was disclosed  in [19].\n",
      "Image Dehazing  \n",
      "The notion of picture dehazing is to recover the real scene radiance of an image from ambient light \n",
      "(A) and the distance from the object to the camera lens  (t) [23].\n",
      "The parameter 𝜌 represents the weight factor that \n",
      "restricts the involvement of brightness value and saturation based on the scenic radiance  [22].\n",
      "Krill Herd algorithm  [13] (KH) is an effective optimization technique that is used to address \n",
      "a large number of issues in bioinspired optimization algorithms.\n",
      "As a result,  the krill \n",
      "density decreases  [20].\n",
      "The krill's density -dependent attraction (growing density) and \n",
      "the search for food are employed as goals, leading to the krill herding around the global minima  [21].\n",
      "References  \n",
      "[1] Kim J.\n",
      "US Patent 5,949,496  \n",
      "[2] Narasimhan SG, Nayar SK.\n",
      "[3] Henry RC, Mahadev S, Urquijo S, Chitwood D. Color perception through atmospheric haze.\n",
      "[4] Liu S, Rahman M, Wong C, Lin S, Jiang G, Kwok N. Dark channel prior based image de -hazing: a review.\n",
      "[5] Lee S, Yun S, Nam J -H, Won CS, Jung S -W. A review on dark channel prior based image dehazing algorithms.\n",
      "[6] Schechner YY, Narasimhan SG, Nayar SK.\n",
      "Appl Opt 2003;42(3):511 –25 \n",
      "[7] Nayar SK, Narasimhan SG.\n",
      "[8] Tan K, Oakley J. Enhancement of color images in poor visibility conditions.\n",
      "[9] Narasimhan SG, Nayar SK.\n",
      "[10] Kopf J, Neubert B, Chen B, Cohen M, Cohen -Or D, Deussen O, et al.\n",
      "[11] He K, Sun J, Tang X. Single image haze removal using dark channel prior.\n",
      "[12] Lee J -S. Digital image enhancement and noise filtering by use of local statistics.\n",
      "Pattern Anal Mach Intell IEEE Trans 1980(2):165 –\n",
      "8 \n",
      "[13] Gandomi, Amir Hossein, and Amir Hossein Alavi.\n",
      "[14] R. C. Gonzalez and R. E. Woods, Digital Image Processing, 3rd ed.\n",
      "Prentice -Hall Inc., Upper Saddle River, NJ, USA, 2006  \n",
      "[15] Y.-T.\n",
      "1 –8, 1997  \n",
      "[16] Y. Wang, Q. Chen, and B. Zhang, “Image enhancement based on equal area dualistic sub -image histogram equalization method,” \n",
      "Consumer Electronics, IEEE  Transactions on, vol.\n",
      "[17] S.-D.\n",
      "[18] K. Sim, C. Tso, and Y. Tan, “Recursive sub -image histogram equalization applied  to gray scale images,” Pattern Recognition \n",
      "Letters, vol.\n",
      "[19] R. Duvar, O. Urhan et al., “Fuzzy fusion based high dynamic range imaging using adaptive histogram separation,” IEEE \n",
      "Transactions on Consumer Electronics,  vol.\n",
      "[20] Darong Huang, Zhou Fang, Ling Zhao, Xiaoyan Chu, \"An improved image clearness algorithm based on dark channel prior\", \n",
      "Proceedings of the 33rd Chinese Control Conference, pp.7350 -7355, 2014.\n",
      "[21] Wei-Jheng Wang, Bo -Hao Chen, Shih -Chia Huang, \"A Novel Visibility Restoration Algorithm for Single Hazy Images\", 2013 \n",
      "IEEE International Conference on Systems, Man, and Cybernetics, pp.847 -851, 2013.\n",
      "[22] Wilawun Punmanee, Teerasit Kasetkasem, Thitiporn Chanwimaluang, Akinori Nishihara, \"Naturalness color enhancement for \n",
      "THEOS images\", 2013 13th International Symposium on Communications and Information Technologies (ISCIT), pp.523 -528, \n",
      "2013.\n",
      "[23] Xue-Hui Wei, Meng Zhao, Zhi -Hai Sun, \"Analysis of color channels effection on image compression distortion\", 2012 \n",
      "International Conference on Wavelet Active Media Technology and Information Processing (ICWAMTIP), pp.157 -160, 2012  \n",
      "[24] Yishu Zhai, Yong Zhang, \"Contrast Restoration for Fog -Degraded Images\", 2009 International Conference on Computational \n",
      "Intelligence and Security, vol.1, pp.619 -623, 2009.\n",
      "[25] Kaiqi Huang, Qiao Wang, Zhenyang Wu, \"Color image enhancement and evaluation algorithm based on human visual system\", \n",
      "2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol.3, pp.iii -721, 2004.\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "from nltk.tokenize import PunktSentenceTokenizer\n",
    "\n",
    "def extract_sentences_with_brackets(text):\n",
    "    # Initialize the tokenizer\n",
    "    tokenizer = PunktSentenceTokenizer()\n",
    "    \n",
    "    # Tokenize the text into sentences\n",
    "    sentences = tokenizer.tokenize(text)\n",
    "    \n",
    "    # Regex pattern to match brackets with numbers and possible text inside\n",
    "    # This pattern attempts to match any sentence with an opening bracket '[' followed by any characters including numbers, \n",
    "    # and closing with a ']' that may or may not be followed by a punctuation mark like a period or comma.\n",
    "    pattern = r'[^.!?]*\\[[^\\]]*\\][^.!?]*[.!?]'\n",
    "    \n",
    "    # Filter sentences that contain brackets\n",
    "    bracket_sentences = [sentence for sentence in sentences if re.search(pattern, sentence)]\n",
    "    \n",
    "    return bracket_sentences\n",
    "\n",
    "# Assuming all_text contains the text extracted from the PDF\n",
    "\n",
    "\n",
    "# Use the function with the text from the PDF\n",
    "bracket_sentences = extract_sentences_with_brackets(all_text)\n",
    "\n",
    "# Print sentences with complex brackets\n",
    "for sentence in bracket_sentences:\n",
    "    print(sentence)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2d22ea3b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracted number list : ['1', '4', '5', '11', '13', '24', '14', '15', '16', '17', '18', '19', '23', '22', '13', '20', '21', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25']\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "res = re.findall(r\"\\[\\s*\\+?(-?\\d+)\\s*\\]\", all_text)\n",
    "print(\"Extracted number list : \" + str(res))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "959ee526",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pandas in c:\\users\\megha\\anaconda3\\anaconda\\lib\\site-packages (1.5.3)Note: you may need to restart the kernel to use updated packages.\n",
      "\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in c:\\users\\megha\\anaconda3\\anaconda\\lib\\site-packages (from pandas) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in c:\\users\\megha\\anaconda3\\anaconda\\lib\\site-packages (from pandas) (2022.7)\n",
      "Requirement already satisfied: numpy>=1.21.0 in c:\\users\\megha\\anaconda3\\anaconda\\lib\\site-packages (from pandas) (1.24.3)\n",
      "Requirement already satisfied: six>=1.5 in c:\\users\\megha\\anaconda3\\anaconda\\lib\\site-packages (from python-dateutil>=2.8.1->pandas) (1.16.0)\n"
     ]
    }
   ],
   "source": [
    "pip install pandas\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2d33b0c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                             Sentence Extracted_Number\n",
      "0   Furthermore, color  distortion is caused by ai...                1\n",
      "1   In a range of applications, clear, high -\\nqua...                4\n",
      "2   In recent \\nyears, various methods for image d...                5\n",
      "3   Multiple image de -hazing [2,6,7], haze \\nremo...               11\n",
      "4   He [11] proposed a technique based on the noti...               13\n",
      "5                                               [13].               24\n",
      "6   As a consequence, the intensity and saturation...               14\n",
      "7   Related Works  \\nThe RGB -HSV conversion [ 14]...               15\n",
      "8   The Mean Brightness Preserving Bi -Histogram E...               16\n",
      "9   In concept, \\nthe Dualistic Sub -Image Histogr...               17\n",
      "10  The technique of Recursive Mean -Separate Hist...               18\n",
      "11  Recursive Sub -Picture Histogram Equalisation ...               19\n",
      "12  The Fuzzy Fusion based High Dynamic Range Imag...               23\n",
      "13  Image Dehazing  \\nThe notion of picture dehazi...               22\n",
      "14  The parameter 𝜌 represents the weight factor t...               13\n",
      "15  Krill Herd algorithm  [13] (KH) is an effectiv...               20\n",
      "16  As a result,  the krill \\ndensity decreases  [...               21\n",
      "17  The krill's density -dependent attraction (gro...                1\n",
      "18                           References  \\n[1] Kim J.                2\n",
      "19  US Patent 5,949,496  \\n[2] Narasimhan SG, Naya...                3\n",
      "20  [3] Henry RC, Mahadev S, Urquijo S, Chitwood D...                4\n",
      "21  [4] Liu S, Rahman M, Wong C, Lin S, Jiang G, K...                5\n",
      "22  [5] Lee S, Yun S, Nam J -H, Won CS, Jung S -W....                6\n",
      "23         [6] Schechner YY, Narasimhan SG, Nayar SK.                7\n",
      "24  Appl Opt 2003;42(3):511 –25 \\n[7] Nayar SK, Na...                8\n",
      "25  [8] Tan K, Oakley J. Enhancement of color imag...                9\n",
      "26                       [9] Narasimhan SG, Nayar SK.               10\n",
      "27  [10] Kopf J, Neubert B, Chen B, Cohen M, Cohen...               11\n",
      "28  [11] He K, Sun J, Tang X. Single image haze re...               12\n",
      "29  [12] Lee J -S. Digital image enhancement and n...               13\n",
      "30  Pattern Anal Mach Intell IEEE Trans 1980(2):16...               14\n",
      "31  [14] R. C. Gonzalez and R. E. Woods, Digital I...               15\n",
      "32  Prentice -Hall Inc., Upper Saddle River, NJ, U...               16\n",
      "33  1 –8, 1997  \\n[16] Y. Wang, Q. Chen, and B. Zh...               17\n",
      "34                                         [17] S.-D.               18\n",
      "35  [18] K. Sim, C. Tso, and Y. Tan, “Recursive su...               19\n",
      "36  [19] R. Duvar, O. Urhan et al., “Fuzzy fusion ...               20\n",
      "37  [20] Darong Huang, Zhou Fang, Ling Zhao, Xiaoy...               21\n",
      "38  [21] Wei-Jheng Wang, Bo -Hao Chen, Shih -Chia ...               22\n",
      "39  [22] Wilawun Punmanee, Teerasit Kasetkasem, Th...               23\n",
      "40  [23] Xue-Hui Wei, Meng Zhao, Zhi -Hai Sun, \"An...               24\n",
      "41  [25] Kaiqi Huang, Qiao Wang, Zhenyang Wu, \"Col...               25\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Assuming `bracket_sentences` is your list of sentences containing brackets\n",
    "# and `res` is your list of extracted numbers\n",
    "\n",
    "# Convert the sentences list and numbers list into a pandas DataFrame\n",
    "df = pd.DataFrame({\n",
    "    'Sentence': bracket_sentences,\n",
    "    'Extracted_Number': pd.Series(res)  # Convert the list of numbers into a pandas Series\n",
    "})\n",
    "\n",
    "# This will create a DataFrame where the first column contains sentences with brackets,\n",
    "# and the second column contains the extracted numbers.\n",
    "# Note: If 'res' contains fewer items than 'bracket_sentences', the excess will be filled with NaN.\n",
    "# If 'res' has more items, those will not be included in the DataFrame due to the length mismatch.\n",
    "\n",
    "print(df)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "489a2c1d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Reference_Number                                     Reference_Text\n",
      "0                 1  In Colorfulness, KH-H outperforms all the exis...\n",
      "1              1999  Kim J.-G.. Color correction device for correct...\n",
      "2                24  Narasimhan SG, Nayar SK. Contrast restoration ...\n",
      "3                 5  Henry RC, Mahadev S, Urquijo S, Chitwood D. Co...\n",
      "4                15  Liu S, Rahman M, Wong C, Lin S, Jiang G, Kwok ...\n",
      "5                23  Lee S, Yun S, Nam J-H, Won CS, Jung S-W. A rev...\n",
      "6              1999  Nayar SK, Narasimhan SG. Vision in bad weather...\n",
      "7                 2  Tan K, Oakley J. Enhancement of color images i...\n",
      "8                 6  Narasimhan SG, Nayar SK. Interactive (de) weat...\n",
      "9                27  Kopf J, Neubert B, Chen B, Cohen M, Cohen-Or D...\n",
      "10               53  He K, Sun J, Tang X. Single image haze removal...\n",
      "11               17  Gandomi, Amir Hossein, and Amir Hossein Alavi....\n",
      "12             1999  Y. Wang, Q. Chen, and B. Zhang, “Image enhance...\n",
      "13                3  S.-D. Chen and A. R. Ramli, “Contrast enhancem...\n",
      "14                7  K. Sim, C. Tso, and Y. Tan, “Recursive sub-ima...\n",
      "15               15  R. Duvar, O. Urhan et al., “Fuzzy fusion based...\n",
      "16               14  Darong Huang, Zhou Fang, Ling Zhao, Xiaoyan Ch...\n",
      "17               13  Wei-Jheng Wang, Bo-Hao Chen, Shih-Chia Huang, ...\n",
      "18               13  Wilawun Punmanee, Teerasit Kasetkasem, Thitipo...\n",
      "19                9  Yishu Zhai, Yong Zhang, \"Contrast Restoration ...\n",
      "20                4  Kaiqi Huang, Qiao Wang, Zhenyang Wu, \"Color im...\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import re\n",
    "from docx import Document\n",
    "\n",
    "# Load the .docx file\n",
    "doc = Document(\"C:\\\\Users\\\\megha\\\\document.docx\")\n",
    "\n",
    "# Initialize an empty list to store references and their numbers\n",
    "references = []\n",
    "reference_numbers = []\n",
    "\n",
    "# Define the regex pattern for extracting references\n",
    "pattern = re.compile(r'([1-9]+)\\..*?$')\n",
    "\n",
    "# Iterate through each paragraph in the document\n",
    "for para in doc.paragraphs:\n",
    "    match = pattern.search(para.text)\n",
    "    if match:\n",
    "        # If a match is found, append the entire reference and the captured number to their respective lists\n",
    "        references.append(para.text)\n",
    "        reference_numbers.append(match.group(1))\n",
    "\n",
    "# Create a DataFrame\n",
    "df_references = pd.DataFrame({\n",
    "    'Reference_Number': reference_numbers,\n",
    "    'Reference_Text': references\n",
    "})\n",
    "\n",
    "# Display or save the DataFrame as needed\n",
    "print(df_references)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5e703769",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Liu S, Rahman M, Wong C, Lin S, Jiang G, Kwok N. Dark channel prior based image de-hazing: a review. In: Information science and technology (ICIST), 2015 5th international conference on. IEEE; 2015. p. 345–50.'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_references.iloc[4,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "cb68253f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting python-docxNote: you may need to restart the kernel to use updated packages.\n",
      "\n",
      "  Obtaining dependency information for python-docx from https://files.pythonhosted.org/packages/5f/d8/6948f7ac00edf74bfa52b3c5e3073df20284bec1db466d13e668fe991707/python_docx-1.1.0-py3-none-any.whl.metadata\n",
      "  Downloading python_docx-1.1.0-py3-none-any.whl.metadata (2.0 kB)\n",
      "Requirement already satisfied: lxml>=3.1.0 in c:\\users\\megha\\anaconda3\\anaconda\\lib\\site-packages (from python-docx) (4.9.2)\n",
      "Requirement already satisfied: typing-extensions in c:\\users\\megha\\anaconda3\\anaconda\\lib\\site-packages (from python-docx) (4.5.0)\n",
      "Downloading python_docx-1.1.0-py3-none-any.whl (239 kB)\n",
      "   ---------------------------------------- 0.0/239.6 kB ? eta -:--:--\n",
      "   ----- --------------------------------- 30.7/239.6 kB 640.0 kB/s eta 0:00:01\n",
      "   ------ -------------------------------- 41.0/239.6 kB 487.6 kB/s eta 0:00:01\n",
      "   ------------- ------------------------- 81.9/239.6 kB 508.4 kB/s eta 0:00:01\n",
      "   --------------- ----------------------- 92.2/239.6 kB 476.3 kB/s eta 0:00:01\n",
      "   ----------------- -------------------- 112.6/239.6 kB 504.4 kB/s eta 0:00:01\n",
      "   ---------------------- --------------- 143.4/239.6 kB 472.1 kB/s eta 0:00:01\n",
      "   ------------------------- ------------ 163.8/239.6 kB 490.7 kB/s eta 0:00:01\n",
      "   --------------------------- ---------- 174.1/239.6 kB 476.3 kB/s eta 0:00:01\n",
      "   -------------------------------- ----- 204.8/239.6 kB 461.0 kB/s eta 0:00:01\n",
      "   -------------------------------------  235.5/239.6 kB 480.3 kB/s eta 0:00:01\n",
      "   -------------------------------------- 239.6/239.6 kB 489.0 kB/s eta 0:00:00\n",
      "Installing collected packages: python-docx\n",
      "Successfully installed python-docx-1.1.0\n"
     ]
    }
   ],
   "source": [
    "pip install python-docx\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "96258803",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Reference_Number                                     Reference_Text\n",
      "0                 1  In Colorfulness, KH-H outperforms all the exis...\n",
      "1              1999  Kim J.-G.. Color correction device for correct...\n",
      "2                24  Narasimhan SG, Nayar SK. Contrast restoration ...\n",
      "3                 5  Henry RC, Mahadev S, Urquijo S, Chitwood D. Co...\n",
      "4                15  Liu S, Rahman M, Wong C, Lin S, Jiang G, Kwok ...\n",
      "5                23  Lee S, Yun S, Nam J-H, Won CS, Jung S-W. A rev...\n",
      "6              1999  Nayar SK, Narasimhan SG. Vision in bad weather...\n",
      "7                 2  Tan K, Oakley J. Enhancement of color images i...\n",
      "8                 6  Narasimhan SG, Nayar SK. Interactive (de) weat...\n",
      "9                27  Kopf J, Neubert B, Chen B, Cohen M, Cohen-Or D...\n",
      "10               53  He K, Sun J, Tang X. Single image haze removal...\n",
      "11               17  Gandomi, Amir Hossein, and Amir Hossein Alavi....\n",
      "12             1999  Y. Wang, Q. Chen, and B. Zhang, “Image enhance...\n",
      "13                3  S.-D. Chen and A. R. Ramli, “Contrast enhancem...\n",
      "14                7  K. Sim, C. Tso, and Y. Tan, “Recursive sub-ima...\n",
      "15               15  R. Duvar, O. Urhan et al., “Fuzzy fusion based...\n",
      "16               14  Darong Huang, Zhou Fang, Ling Zhao, Xiaoyan Ch...\n",
      "17               13  Wei-Jheng Wang, Bo-Hao Chen, Shih-Chia Huang, ...\n",
      "18               13  Wilawun Punmanee, Teerasit Kasetkasem, Thitipo...\n",
      "19                9  Yishu Zhai, Yong Zhang, \"Contrast Restoration ...\n",
      "20                4  Kaiqi Huang, Qiao Wang, Zhenyang Wu, \"Color im...\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import re\n",
    "from docx import Document\n",
    "\n",
    "# Load the .docx file\n",
    "doc = Document(\"C:\\\\Users\\\\megha\\\\document.docx\")\n",
    "\n",
    "# Initialize an empty list to store references and their numbers\n",
    "references = []\n",
    "reference_numbers = []\n",
    "\n",
    "# Define the regex pattern for extracting references\n",
    "pattern = re.compile(r'([1-9]+)\\..*?$')\n",
    "\n",
    "# Iterate through each paragraph in the document\n",
    "for para in doc.paragraphs:\n",
    "    match = pattern.search(para.text)\n",
    "    if match:\n",
    "        # If a match is found, append the entire reference and the captured number to their respective lists\n",
    "        references.append(para.text)\n",
    "        reference_numbers.append(match.group(1))\n",
    "\n",
    "# Create a DataFrame\n",
    "df_references = pd.DataFrame({\n",
    "    'Reference_Number': reference_numbers,\n",
    "    'Reference_Text': references\n",
    "})\n",
    "\n",
    "# Display or save the DataFrame as needed\n",
    "print(df_references)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4ca16112",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter the title of the research paper: Narasimhan SG, Nayar SK. Contrast restoration of weather degraded images. Pattern Anal Mach Intell IEEE Trans 2003a\n",
      "DOI: 10.1109/tpami.2003.1201821\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "def get_doi_from_title(title):\n",
    "    base_url = \"https://api.crossref.org/works\"\n",
    "    params = {\"query.bibliographic\": title}\n",
    "    \n",
    "    try:\n",
    "        response = requests.get(base_url, params=params)\n",
    "        response.raise_for_status()\n",
    "        data = response.json()\n",
    "        items = data.get(\"message\", {}).get(\"items\", [])\n",
    "        \n",
    "        if items:\n",
    "            # Extract DOI from the first item\n",
    "            doi = items[0].get(\"DOI\")\n",
    "            return doi\n",
    "        else:\n",
    "            print(\"No DOI found for the given title.\")\n",
    "            return None\n",
    "    except requests.exceptions.RequestException as e:\n",
    "        print(\"Error:\", e)\n",
    "        return None\n",
    "\n",
    "# Example usage:\n",
    "title = input(\"Enter the title of the research paper: \")\n",
    "doi = get_doi_from_title(title)\n",
    "if doi:\n",
    "    print(\"DOI:\", doi)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "fdf4a8e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter the DOI of the research paper: 10.1109/tpami.2003.1201821\n",
      "Abstract: Images of outdoor scenes captured in bad weather suffer from poor contrast. Under bad weather conditions, the light reaching a camera is severely scattered by the atmosphere. The resulting decay in contrast varies across the scene and is exponential in the depths of scene points. Therefore, traditional space invariant image processing techniques are not sufficient to remove weather effects from images. We present a physics-based model that describes the appearances of scenes in uniform bad weather conditions. Changes in intensities of scene points under different weather conditions provide simple constraints to detect depth discontinuities in the scene and also to compute scene structure. Then, a fast algorithm to restore scene contrast is presented. In contrast to previous techniques, our weather removal algorithm does not require any a priori scene structure, distributions of scene reflectances, or detailed knowledge about the particular weather condition. All the methods described in this paper are effective under a wide range of weather conditions including haze, mist, fog, and conditions arising due to other aerosols. Further, our methods can be applied to gray scale, RGB color, multispectral and even IR images. We also extend our techniques to restore contrast of scenes with moving objects, captured using a video camera.\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "def get_abstract_from_doi(doi):\n",
    "    base_url = f\"https://api.semanticscholar.org/v1/paper/{doi}\"\n",
    "    \n",
    "    try:\n",
    "        response = requests.get(base_url)\n",
    "        response.raise_for_status()\n",
    "        data = response.json()\n",
    "        abstract = data.get(\"abstract\")\n",
    "        \n",
    "        if abstract:\n",
    "            return abstract\n",
    "        else:\n",
    "            print(\"No abstract found for the given DOI.\")\n",
    "            return None\n",
    "    except requests.exceptions.RequestException as e:\n",
    "        print(\"Error:\", e)\n",
    "        return None\n",
    "\n",
    "# Example usage:\n",
    "doi = input(\"Enter the DOI of the research paper: \")\n",
    "abstract = get_abstract_from_doi(doi)\n",
    "if abstract:\n",
    "    print(\"Abstract:\", abstract)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "eaadb3a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "81b627dc9174487689aed189df73b7c9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/399 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\megha\\anaconda3\\anaconda\\Lib\\site-packages\\huggingface_hub\\file_download.py:149: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\\Users\\megha\\.cache\\huggingface\\hub\\models--sentence-transformers--bert-base-nli-mean-tokens. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.\n",
      "To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development\n",
      "  warnings.warn(message)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a3ccf3d244ff40ad8ee89e239530d97b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/625 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cd12247cfc06499fb1724e9e5f7b1a1e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a72a5a7d057644529451373c88899706",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f694ae7e730b432ca625530fbf49cc6e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "added_tokens.json:   0%|          | 0.00/2.00 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc09510dbd2c491ebf637cd274e69750",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/112 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "11cbcaa6a0ed4c9a9d29778599051eb7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/438M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "The size of tensor a (4414) must match the size of tensor b (512) at non-singleton dimension 1",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[17], line 42\u001b[0m\n\u001b[0;32m     39\u001b[0m document_text \u001b[38;5;241m=\u001b[39m docx2txt\u001b[38;5;241m.\u001b[39mprocess(document_path)\n\u001b[0;32m     41\u001b[0m abstract_embedding \u001b[38;5;241m=\u001b[39m get_embedding(abstract, tokenizer, model)\n\u001b[1;32m---> 42\u001b[0m document_embedding \u001b[38;5;241m=\u001b[39m get_embedding(document_text, tokenizer, model)\n\u001b[0;32m     44\u001b[0m similarity_score \u001b[38;5;241m=\u001b[39m cosine_similarity(abstract_embedding\u001b[38;5;241m.\u001b[39msqueeze(), document_embedding\u001b[38;5;241m.\u001b[39msqueeze())\n\u001b[0;32m     45\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mSimilarity Score:\u001b[39m\u001b[38;5;124m\"\u001b[39m, similarity_score\u001b[38;5;241m.\u001b[39mitem())\n",
      "Cell \u001b[1;32mIn[17], line 17\u001b[0m, in \u001b[0;36mget_embedding\u001b[1;34m(text, tokenizer, model)\u001b[0m\n\u001b[0;32m     15\u001b[0m \u001b[38;5;66;03m# Compute token embeddings\u001b[39;00m\n\u001b[0;32m     16\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m torch\u001b[38;5;241m.\u001b[39mno_grad():\n\u001b[1;32m---> 17\u001b[0m     model_output \u001b[38;5;241m=\u001b[39m model(\u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mencoded_input)\n\u001b[0;32m     18\u001b[0m \u001b[38;5;66;03m# Perform mean pooling to get sentence embeddings\u001b[39;00m\n\u001b[0;32m     19\u001b[0m embeddings \u001b[38;5;241m=\u001b[39m model_output\u001b[38;5;241m.\u001b[39mlast_hidden_state\n",
      "File \u001b[1;32m~\\anaconda3\\anaconda\\Lib\\site-packages\\torch\\nn\\modules\\module.py:1518\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m   1516\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_compiled_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[0;32m   1517\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m-> 1518\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
      "File \u001b[1;32m~\\anaconda3\\anaconda\\Lib\\site-packages\\torch\\nn\\modules\\module.py:1527\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m   1522\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[0;32m   1523\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[0;32m   1524\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks\n\u001b[0;32m   1525\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[0;32m   1526\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[1;32m-> 1527\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m forward_call(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[0;32m   1529\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m   1530\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "File \u001b[1;32m~\\anaconda3\\anaconda\\Lib\\site-packages\\transformers\\models\\bert\\modeling_bert.py:1006\u001b[0m, in \u001b[0;36mBertModel.forward\u001b[1;34m(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)\u001b[0m\n\u001b[0;32m    999\u001b[0m \u001b[38;5;66;03m# Prepare head mask if needed\u001b[39;00m\n\u001b[0;32m   1000\u001b[0m \u001b[38;5;66;03m# 1.0 in head_mask indicate we keep the head\u001b[39;00m\n\u001b[0;32m   1001\u001b[0m \u001b[38;5;66;03m# attention_probs has shape bsz x n_heads x N x N\u001b[39;00m\n\u001b[0;32m   1002\u001b[0m \u001b[38;5;66;03m# input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]\u001b[39;00m\n\u001b[0;32m   1003\u001b[0m \u001b[38;5;66;03m# and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]\u001b[39;00m\n\u001b[0;32m   1004\u001b[0m head_mask \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mget_head_mask(head_mask, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mconfig\u001b[38;5;241m.\u001b[39mnum_hidden_layers)\n\u001b[1;32m-> 1006\u001b[0m embedding_output \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39membeddings(\n\u001b[0;32m   1007\u001b[0m     input_ids\u001b[38;5;241m=\u001b[39minput_ids,\n\u001b[0;32m   1008\u001b[0m     position_ids\u001b[38;5;241m=\u001b[39mposition_ids,\n\u001b[0;32m   1009\u001b[0m     token_type_ids\u001b[38;5;241m=\u001b[39mtoken_type_ids,\n\u001b[0;32m   1010\u001b[0m     inputs_embeds\u001b[38;5;241m=\u001b[39minputs_embeds,\n\u001b[0;32m   1011\u001b[0m     past_key_values_length\u001b[38;5;241m=\u001b[39mpast_key_values_length,\n\u001b[0;32m   1012\u001b[0m )\n\u001b[0;32m   1013\u001b[0m encoder_outputs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mencoder(\n\u001b[0;32m   1014\u001b[0m     embedding_output,\n\u001b[0;32m   1015\u001b[0m     attention_mask\u001b[38;5;241m=\u001b[39mextended_attention_mask,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m   1023\u001b[0m     return_dict\u001b[38;5;241m=\u001b[39mreturn_dict,\n\u001b[0;32m   1024\u001b[0m )\n\u001b[0;32m   1025\u001b[0m sequence_output \u001b[38;5;241m=\u001b[39m encoder_outputs[\u001b[38;5;241m0\u001b[39m]\n",
      "File \u001b[1;32m~\\anaconda3\\anaconda\\Lib\\site-packages\\torch\\nn\\modules\\module.py:1518\u001b[0m, in \u001b[0;36mModule._wrapped_call_impl\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m   1516\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_compiled_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)  \u001b[38;5;66;03m# type: ignore[misc]\u001b[39;00m\n\u001b[0;32m   1517\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m-> 1518\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_call_impl(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
      "File \u001b[1;32m~\\anaconda3\\anaconda\\Lib\\site-packages\\torch\\nn\\modules\\module.py:1527\u001b[0m, in \u001b[0;36mModule._call_impl\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m   1522\u001b[0m \u001b[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001b[39;00m\n\u001b[0;32m   1523\u001b[0m \u001b[38;5;66;03m# this function, and just call forward.\u001b[39;00m\n\u001b[0;32m   1524\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m (\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_forward_pre_hooks\n\u001b[0;32m   1525\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_backward_pre_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_backward_hooks\n\u001b[0;32m   1526\u001b[0m         \u001b[38;5;129;01mor\u001b[39;00m _global_forward_hooks \u001b[38;5;129;01mor\u001b[39;00m _global_forward_pre_hooks):\n\u001b[1;32m-> 1527\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m forward_call(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[0;32m   1529\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m   1530\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "File \u001b[1;32m~\\anaconda3\\anaconda\\Lib\\site-packages\\transformers\\models\\bert\\modeling_bert.py:238\u001b[0m, in \u001b[0;36mBertEmbeddings.forward\u001b[1;34m(self, input_ids, token_type_ids, position_ids, inputs_embeds, past_key_values_length)\u001b[0m\n\u001b[0;32m    236\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mposition_embedding_type \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mabsolute\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n\u001b[0;32m    237\u001b[0m     position_embeddings \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mposition_embeddings(position_ids)\n\u001b[1;32m--> 238\u001b[0m     embeddings \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m position_embeddings\n\u001b[0;32m    239\u001b[0m embeddings \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mLayerNorm(embeddings)\n\u001b[0;32m    240\u001b[0m embeddings \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdropout(embeddings)\n",
      "\u001b[1;31mRuntimeError\u001b[0m: The size of tensor a (4414) must match the size of tensor b (512) at non-singleton dimension 1"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModel\n",
    "import docx2txt\n",
    "\n",
    "# Function to load model and tokenizer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f5f8a61b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting docx2txt\n",
      "  Downloading docx2txt-0.8.tar.gz (2.8 kB)\n",
      "  Preparing metadata (setup.py): started\n",
      "  Preparing metadata (setup.py): finished with status 'done'\n",
      "Building wheels for collected packages: docx2txt\n",
      "  Building wheel for docx2txt (setup.py): started\n",
      "  Building wheel for docx2txt (setup.py): finished with status 'done'\n",
      "  Created wheel for docx2txt: filename=docx2txt-0.8-py3-none-any.whl size=3973 sha256=89de490417e4f1232d0147119f4759765b04d53c55b4861f8b12c5beed707d79\n",
      "  Stored in directory: c:\\users\\megha\\appdata\\local\\pip\\cache\\wheels\\0f\\0e\\7a\\3094a4ceefe657bff7e12dd9592a9d5b6487ef4338ace0afa6\n",
      "Successfully built docx2txt\n",
      "Installing collected packages: docx2txt\n",
      "Successfully installed docx2txt-0.8\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install docx2txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f5e5c501",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity Score: 0.8118084669113159\n"
     ]
    }
   ],
   "source": [
    "def load_model():\n",
    "    tokenizer = AutoTokenizer.from_pretrained(\"sentence-transformers/bert-base-nli-mean-tokens\")\n",
    "    model = AutoModel.from_pretrained(\"sentence-transformers/bert-base-nli-mean-tokens\")\n",
    "    return tokenizer, model\n",
    "\n",
    "# Function to preprocess and generate embeddings\n",
    "def load_model():\n",
    "    tokenizer = AutoTokenizer.from_pretrained(\"sentence-transformers/bert-base-nli-mean-tokens\")\n",
    "    model = AutoModel.from_pretrained(\"sentence-transformers/bert-base-nli-mean-tokens\")\n",
    "    return tokenizer, model\n",
    "\n",
    "# Function to preprocess and generate embeddings\n",
    "def get_embedding(text, tokenizer, model, max_length=512):\n",
    "    # Split the text into chunks of `max_length`\n",
    "    tokenized_text = tokenizer.encode_plus(text, add_special_tokens=True, truncation=True, max_length=max_length)\n",
    "    total_length = len(tokenized_text['input_ids'])\n",
    "    chunks = [text[i:i + max_length] for i in range(0, total_length, max_length)]\n",
    "\n",
    "    all_embeddings = []\n",
    "    for chunk in chunks:\n",
    "        # Encode the chunk using tokenizer\n",
    "        encoded_input = tokenizer(chunk, padding=True, truncation=True, return_tensors='pt', max_length=max_length)\n",
    "        # Compute token embeddings\n",
    "        with torch.no_grad():\n",
    "            model_output = model(**encoded_input)\n",
    "        # Perform mean pooling to get sentence embeddings\n",
    "        embeddings = model_output.last_hidden_state\n",
    "        attention_mask = encoded_input['attention_mask']\n",
    "        mask_expansion = attention_mask.unsqueeze(-1).expand(embeddings.size()).float()\n",
    "        sum_embeddings = torch.sum(embeddings * mask_expansion, 1)\n",
    "        sum_mask = torch.clamp(mask_expansion.sum(1), min=1e-9)\n",
    "        all_embeddings.append(sum_embeddings / sum_mask)\n",
    "\n",
    "    # Average the embeddings from all chunks\n",
    "    if all_embeddings:\n",
    "        embeddings_avg = torch.mean(torch.stack(all_embeddings), dim=0)\n",
    "        return embeddings_avg\n",
    "    else:\n",
    "        # Return zero tensor if no valid embeddings were computed\n",
    "        return torch.zeros(model.config.hidden_size)\n",
    "\n",
    "\n",
    "# Function to compute cosine similarity\n",
    "def cosine_similarity(tensor1, tensor2):\n",
    "    dot_product = torch.dot(tensor1, tensor2)\n",
    "    norm_tensor1 = tensor1.norm()\n",
    "    norm_tensor2 = tensor2.norm()\n",
    "    return dot_product / (norm_tensor1 * norm_tensor2)\n",
    "\n",
    "# Main execution\n",
    "tokenizer, model = load_model()\n",
    "\n",
    "abstract = \"\"\"Images of outdoor scenes captured in bad weather suffer from poor contrast. Under bad weather conditions, the light reaching a camera is severely scattered by the atmosphere. The resulting decay in contrast varies across the scene and is exponential in the depths of scene points. Therefore, traditional space invariant image processing techniques are not sufficient to remove weather effects from images. We present a physics-based model that describes the appearances of scenes in uniform bad weather conditions. Changes in intensities of scene points under different weather conditions provide simple constraints to detect depth discontinuities in the scene and also to compute scene structure. Then, a fast algorithm to restore scene contrast is presented. In contrast to previous techniques, our weather removal algorithm does not require any a priori scene structure, distributions of scene reflectances, or detailed knowledge about the particular weather condition. All the methods described in this paper are effective under a wide range of weather conditions including haze, mist, fog, and conditions arising due to other aerosols. Further, our methods can be applied to gray scale, RGB color, multispectral and even IR images. We also extend our techniques to restore contrast of scenes with moving objects, captured using a video camera.\"\"\"\n",
    "document_path = \"C:\\\\Users\\\\megha\\\\document.docx\"\n",
    "\n",
    "document_text = docx2txt.process(document_path)\n",
    "\n",
    "abstract_embedding = get_embedding(abstract, tokenizer, model)\n",
    "document_embedding = get_embedding(document_text, tokenizer, model)\n",
    "\n",
    "similarity_score = cosine_similarity(abstract_embedding.squeeze(), document_embedding.squeeze())\n",
    "print(\"Similarity Score:\", similarity_score.item())\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3245ee84",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}