Polyglot programming using Jenkins

 Facility for languages develops when one does not squander existing opportunities to code. That is what I think.

Jenkins, the CI enabler supports a few languages like Python and Groovy. The Python package I used to make the Rest API calls is ‘Python Jenkins’.It is interesting to note that run_script executes Groovy code.

I didn’t test it exactly when the Unix server runs out of disk space but assumed the text from the console output will match.Moreover the encryption routine works as expected but the decryption function doesn’t work. It seems that since I call the Rest API there could be a encryption/decryption key mismatch.

'''
Created on Oct 12, 2016

@author: Mohan Radhakrishnan

This python module gets the console output of the latest
build and if the text 'No space left on device' is found in
the output it sends a mail.
I've taken liberties with the 'functional paradigm'

'''
import smtplib
import jenkins
import os
def main():

overrideenvironmentvariables()

server = jenkins.Jenkins('http://localhost:8080', username='Mohan', password='Welcome1')

notifydisaster(server)

'''
Notify
'''
def notifydisaster( server ):
print( getconsoleoutput(server) )
name,buildnumber,consoleoutput = getconsoleoutput(server)
if (consoleoutput.find("Caused by: java.io.IOException: No space left on device") != -1):
print("Caused by: java.io.IOException: No space left on device")
sendmail( name,buildnumber )

'''
Notify
Password Encryption/decryption code has to be tested and used
'''
def sendmail(name,buildnumber):
smtp = smtplib.SMTP('smtp.gmail.com', 587)
smtp.ehlo()
smtp.starttls()
smtp.login("x.y@z.com","Password")
smtp.sendmail('x.y@z.com', 'x.y@z.com', 'Subject: No space left on device\n \
Job ' + name + ' Build ' + str(buildnumber) + ' fails due to lack of disk space')

'''
Get the console output of the particular
Job's build
'''
def getconsoleoutput(server):
information = getJobName(server)
if information:
return information[ 0 ]['name'] ,getlastjobDetails(server),server.get_build_console_output(information[ 0 ]['name'], getlastjobDetails(server))

'''
Get Job and other details
and filter the Job we are interested in
'''
def getJobName(server):
jobs = server.get_all_jobs(0)
filtercriterion = ['CITestPipeline']

return list(filter( lambda d: d['fullname'] in filtercriterion, jobs))

'''
Get Job and other details
Return '0' as the build number assuming
it signifies that there is no such build number
'''

def getlastjobDetails(server):
information = getJobName(server)
if information:
last_build_number = server.get_job_info(information[ 0 ]['name'])['lastCompletedBuild']['number']
return last_build_number
else:
return 0

'''
Attempt here to encrypt Passwords using Jenkins' key
Not tested properly
'''
def encrypt(server ):
value = server.run_script("""
secret = hudson.util.Secret.fromString("Password")
println secret.getEncryptedValue()
println secret.getPlainText()
""")
print (value)

def decrypt(server ):
decryptedvalue = server.run_script("""
secret = hudson.util.Secret.fromString("aiJREkuBjWHX9UWIyhEzwnnAJReuZnQVEtUr0KgvXKg")
println hudson.util.Secret.toString(secret)
""")
print (decryptedvalue)
return decryptedvalue
'''
Override this proxy setting as we don't
need it and it causes an error.
'''
def overrideenvironmentvariables():
os.environ["HTTP_PROXY"] = ''

if __name__=="__main__":
main()

Parsing HTML using BeautifulSoup

This Python code that parses HTML seems to truncate the tags when I print it. I am attempting to check for the presence of the ID attribute in the tags. The code just iterates over all tags and it does not specifically look for a HTML control. It just matches the opening and closing tag arbitratrily. I am still working on it and will update it.

D:\Python Data Analytics\view.html
   No                                                Tag
0   1  <div class="panel-collapse collapse" id="activ...
1   1  <select class="selectpicker " id="condition1...
2   1  <select class="selectpicker " id="condition2...
3   1  <select class="selectpicker " id="condition3...
4   1  <select class="selectpicker " id="condition4...
5   1  <select class="selectpicker " id="condition5...
6   1  <input class="btn xbtn-primary save" id="ApS...
7   1  <input class="btn btn-link" id="Cancel" name...

from bs4 import BeautifulSoup as bs
import sys
import os
import pandas as pd
import fnmatch

class Parse:
 
    def __init__(self):
        self.parse()


    def parse(self):
        
        pd.options.display.max_colwidth = 0
        try:
            path = "D:\\Python Data Analytics\\"
            
            f = open('D:\python\\report.html','w')

 
             #Pattern to be matched
            includes = ['*.html']
        
            for root, subFolders, files in os.walk(path):
                 for extensions in includes:
                     
                    for infile in fnmatch.filter(files, extensions):
                            soup = bs(open( path + infile, "r").read())
                                        
                            data = soup.findAll(True,{'id':True})
                            
                            df = pd.DataFrame(columns=[
                                                       'ID',
                                                       'Tag'])

                            idattributes = []
                            duplicates = [] 
                            
                            for attribute in data:
                                idTag = attribute.find('id')
                                att = attribute.attrs
                                idattributes.append(att['id'])
                                df = df.append(pd.DataFrame( [dict(
                                                                   ID=att['id'],
                                                                   Tag=attribute)] ),
                                                                   ignore_index=True)
                            s = set()
                            duplicates = set(x for x in idattributes if x in s or s.add(x))  
                                                              
                            data1 = soup.findAll(attrs={'id': None})
                            df1 = pd.DataFrame(columns=[
                                                       
                                                       'Tag'])
            
                            missingid = {} 
                            count = 0
                            for attribute in data1:
                                    missingid.update({count: attribute})
                                    df1 = df1.append(pd.DataFrame( [dict(
                                                                   Tag=attribute)] ),
                                                                   ignore_index=True)
                                    count = count + 1
                                    
                            df2 = pd.DataFrame(missingid.items())
                            html5report = df
                            print df2
                            html5report1 = df2
                            
                            table = ""
                            table += '<table>'
                            for element in duplicates:
                                table += '  <tr>'
                                table += '    <td>' + element + '</td>'
                                table += '  </tr>'
                            table += '</table>'
                            
                            html5report1 = html5report1.to_html().replace('<table border="1" class="dataframe">','<table class="table table-striped">')
                            html5report = html5report.to_html().replace('<table border="1" class="dataframe">','<table class="table table-striped">')
                            htmlreporter = '''
            							<html>
                							<head>
                    						<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css">
                    						<style>body{ margin:0 100; background:whitesmoke; }</style>
                							</head>
                                            <body>
                                            <h1>HTML 5 Report</h1>
                                            <h2>''' + infile + '''</h2>
            
                                            <h3>Tags with ID present</h3>
                   								''' + html5report + '''
                                            <h3>Tags with ID not present</h3>
                                                ''' + html5report1 + '''
                                            <h3>Possible Duplicates</h3>
                                                ''' + table + '''
                							</body>
            				</html>'''
                            f.write(htmlreporter)
            f.close()    
                            
        except IOError as e:
            print "I/O error({0}): {1}".format(e.errno, e.strerror)
        except Exception, err:
            print "Unexpected error:", sys.exc_info()[0]
 

if __name__ == '__main__':
    instance = Parse()

Matrix multiplication

This is simplified code that multiplies two matrices – a and b – using data stored in a file.

["a", 0, 0, 63]
["a", 0, 1, 45]
["a", 0, 2, 93]
["a", 0, 3, 32]
["a", 0, 4, 49]
["a", 1, 0, 33]
["a", 1, 3, 26]
["a", 1, 4, 95]
["a", 2, 0, 25]
["a", 2, 1, 11]
["a", 2, 3, 60]
["a", 2, 4, 89]
["a", 3, 0, 24]
["a", 3, 1, 79]
["a", 3, 2, 24]
["a", 3, 3, 47]
["a", 3, 4, 18]
["a", 4, 0, 7]
["a", 4, 1, 98]
["a", 4, 2, 96]
["a", 4, 3, 27]
["b", 0, 0, 63]
["b", 0, 1, 18]
["b", 0, 2, 89]
["b", 0, 3, 28]
["b", 0, 4, 39]
["b", 1, 0, 59]
["b", 1, 1, 76]
["b", 1, 2, 34]
["b", 1, 3, 12]
["b", 1, 4, 6]
["b", 2, 0, 30]
["b", 2, 1, 52]
["b", 2, 2, 49]
["b", 2, 3, 3]
["b", 2, 4, 95]
["b", 3, 0, 77]
["b", 3, 1, 75]
["b", 3, 2, 85]
["b", 4, 1, 46]
["b", 4, 2, 33]
["b", 4, 3, 69]
["b", 4, 4, 88]

I loaded this data in a dict so that the keys are the tuples(row,column).

      for x in sorted(matrixa.items()):
            print x

((0, 0), 63)
((0, 1), 45)
((0, 2), 93)
((0, 3), 32)
((0, 4), 49)
((1, 0), 33)
((1, 3), 26)
((1, 4), 95)
((2, 0), 25)
((2, 1), 11)
((2, 3), 60)
((2, 4), 89)
((3, 0), 24)
((3, 1), 79)
((3, 2), 24)
((3, 3), 47)
((3, 4), 18)
((4, 0), 7)
((4, 1), 98)
((4, 2), 96)
((4, 3), 27)

It is a sparse matrix. So all the values may not be there. The code has to check that.

     l = []

      for i in range(0,5):
        for j in range(0,5):
            for k in range(0,5):
               if (i,k) in matrixa:
                   x = matrixa[(i,k)]
               else:
                   x = 0
               if (k,j) in matrixb:
                   y = matrixb[(k,j)]
               else:
                   y = 0
               z += x * y
            l.append([i,j,z])
            z = 0
     for i in l:
        print i

The result is this.

[0, 0, 11878]
[0, 1, 14044]
[0, 2, 16031]
[0, 3, 5964]
[0, 4, 15874]
[1, 0, 4081]
[1, 1, 6914]
[1, 2, 8282]
[1, 3, 7479]
[1, 4, 9647]
[2, 0, 6844]
[2, 1, 9880]
[2, 2, 10636]
[2, 3, 6973]
[2, 4, 8873]
[3, 0, 10512]
[3, 1, 12037]
[3, 2, 10587]
[3, 3, 2934]
[3, 4, 5274]
[4, 0, 11182]
[4, 1, 14591]
[4, 2, 10954]
[4, 3, 1660]
[4, 4, 9981]

Filter a Python dictionary

((u'Valjean', u'MmeDeR'), (u'MmeDeR', u'Valjean'))
((u'Valjean', u'Montparnasse'), (u'Montparnasse', u'Valjean'))
((u'Myriel', u'Count'), (u'Count', u'Myriel'))
((u'Valjean', u'Judge'), (u'Judge', u'Valjean'))
((u'Napoleon', u'Myriel'), (u'Myriel', u'Napoleon'))
((u'Myriel', u'Valjean'), (u'Valjean', u'Myriel'))
((u'Champtercier', u'Myriel'), (u'Myriel', u'Champtercier'))
((u'Myriel', u'OldMan'), (u'OldMan', u'Myriel'))
((u'Valjean', u'Woman1'), (u'Woman1', u'Valjean'))
((u'Valjean', u'Babet'), (u'Babet', u'Valjean'))
((u'MlleBaptistine', u'Valjean'), (u'Valjean', u'MlleBaptistine'))
((u'MmeMagloire', u'Myriel'), (u'Myriel', u'MmeMagloire'))
((u'Valjean', u'Labarre'), (u'Labarre', u'Valjean'))
((u'Valjean', u'Woman2'), (u'Woman2', u'Valjean'))
((u'Myriel', u'Geborand'), (u'Geborand', u'Myriel'))
((u'Valjean', u'Marguerite'), (u'Marguerite', u'Valjean'))
((u'MlleBaptistine', u'MmeMagloire'), (u'MmeMagloire', u'MlleBaptistine'))
((u'Valjean', u'Isabeau'), (u'Isabeau', u'Valjean'))
((u'Valjean', u'Simplice'), (u'Simplice', u'Valjean'))
((u'Valjean', u'Gillenormand'), (u'Gillenormand', u'Valjean'))
((u'Valjean', u'MlleGillenormand'), (u'MlleGillenormand', u'Valjean'))
((u'Myriel', u'Champtercier'), (u'Champtercier', u'Myriel'))
((u'Valjean', u'MmeMagloire'), (u'MmeMagloire', u'Valjean'))
((u'Valjean', u'Fantine'), (u'Fantine', u'Valjean'))
((u'MlleBaptistine', u'Myriel'), (u'Myriel', u'MlleBaptistine'))
((u'Valjean', u'Myriel'), (u'Myriel', u'Valjean'))
((u'Valjean', u'Cosette'), (u'Cosette', u'Valjean'))

I am used to the verbosity of Java. So Python is like a breath of fresh air.
But this Python code stumped me for the better part of two days.

I was trying to filter the keys in the dictionary shown above based on its own values. One of the examples is shown in a different color. So if the key matches a value then that dictionary row with that particular key should be removed. The row with the matching value is untouched.

It is literally one line of Python.

    for v in friends.iteritems():
        print v
    result = [k for k, v in friends.iteritems() if v not in friends.keys()]
    for v in result:
        print v