Thursday, December 22, 2016

Create #VirtualPrivateCloud, NAT Instance and NAT Gateways on @AWSCloud

Create a Virtual Private Cloud, NAT instance and the new NAT Gatweay ... and making it all work.
This is a YouTube playlist of three videos.


Saturday, December 17, 2016

Cheat sheet to create a #VPC and Subnets on @AWSCloud

One of the critical things to remember for working with a AWS VPC is creating and using it. I had hard time remembering how to do it, so, I wrote down a cheat sheet for myself. 

If anyone wants to follow along, just navigate to the VPC page on the AWS Console and start with 'Create VPC' button. Please note that this may cost some dollars if you are not on the free tier. If you are on the free tier and make mistakes, it may cost some dollars.

In the steps below, we will be creating the following on a new VPC:
  1. An internet gateway
  2. One public subnet with routes for accessibility from the internet
  3. One private subnet without any routes
  4. One EC2 web server with Apache installed in it and serving a sample html page - using the public subnet.
  5. One EC2 server with the private subnet and security group that allows access to resources running on the public subnet only. 

Create VPC

  1. Name tag: myVPC
  2. CIDR Block: 10.0.0.0/16
  3. Tenancy: default (Must have default. Otherwise, it will get very expensive quite fast.)

Create subnet for public

  1. Name tag: 10.0.1.0 - us-west-1a
  2. VPC: myVPC
  3. Availability zone: us-west-1a
  4. CIDR Block: 10.0.1.0/24
  5. Select the public subnet from the VPC console
    1. Subnet Actions drop down - Modify Auto Assign IP - Enable auto-assign public IPv4 address.
The last step ensures that your public web server will be accessible on the internet with an automatically assigned public IP address.

Create another subnet for private

  1. Name tag: 10.0.2.0 - use-west-1b
  2. VPC: myVPC
  3. Availability zone: us-west-1b
  4. CIDR Block: 10.0.2.0/24

Create Internet Gateway

  1. Name tag: myIGW
  2. Then attach to VPC: myVPC

Create Route Tables for public

  1. Name tag: myPublicRoute
  2. VPC: myVPC
  3. Routes tab: Edit
    1. Description: 0.0.0.0/0
    2. Target: myIGW (it may look different than the keyword myIGW, but it is the only one that will be selectable.)
  4. Subnet associations: Edit
    1. Checkbox: us-west-1a subnet

Create EC2 instance for public route

  1. Network: myVPC
  2. Subnet: 10.0.1.0 - us-west-1a
  3. Advanced details:
    1. Install httpd, update, start httpd and configure httpd to start on server start.
      1. yum install httpd -y && yum update -y && service httpd start && chkconfig httpd on
    2. Create a simple index.html
      1. echo “<html><h1></h1></html>” > /var/www/html/index.html
  4. Tag instance: MyWebServer
  5. New security group: myWebSG
    1. ssh 0.0.0.0/0
    2. http 0.0.0.0/0
  6. Launch
  7. Get public IP address and open in browser

Create EC2 instance for private route
  1. Network: myVPC
  2. Subnet: 10.0.2.0 - us-west-1b
  3. Tag instance: MyWebServer
  4. New security group: myWebSG
    1. ssh 10.0.1.0/24
    2. http 10.0.1.0/24
    3. All ICMP 10.0.1.0/24
  5. Launch - no public ip is provided
  6. Make a note of the private ip address

Verify

SSH to public EC2 instance using your pem key.
  1. Copy the contents of your pem key and save it on the public EC2 instance.
  2. From the public EC2 instance, SSH to the private EC2 instance using the copied pem key using the private ip address.

The private EC2 instance is accessible only from the public web server. It is not directly accessible from the internet. In fact, you cannot SSH to the private EC2 instance because there is no public IP address assigned to the instance.





Friday, November 11, 2016

#AWS Developer Notes - Part 1

Region is geographic area.

Each region has availability zones - different data centers.

Edge Locations - CDN end points for CloudFront.

AWS Platform


1.0 Networking

VPC (Virtual Private Cloud)

Virtual Data Center in your AWS account. Logically separate network.

Direct Connect

A way of connecting into AWS environment without using an internet connection. 

Route53

DNS service. 53 is the default DNS port.


2.0 Compute

EC2

Virtual server provisioned in a few seconds or minutes.

EC2 Container Service

Scalable fast container management service to manage and run Docker containers.

Elastic Beanstalk

Easy to use service to deploy and scale apps and services. Designed for developers to upload their code and beanstalk will inspect the code and provision the resources underneath.
(Covered in detail for DevOps exam)

Lambda

Most powerful AWS service. Lets you run code without provisioning or managing servers. Pay for the compute time only. Pay only for execution time.

3.0 Storage

S3

Object based storage. Place to store your flat files in the cloud. Secure and highly scalable and only pay for storage that you actually use.

Cloud Front

CDN. Integrates with other AWS products like S3. Different edge locations around the world to cache your files.

Glacier

Secure, durable and low cost storage solution. Anything stored in Glacier can take up to 4 hours to access it. It is good for long-term storage.

EFS

Elastic File System - file storage for EC2. Necessarily NFS. Block level rather than object level.

Snowball

Amazon's import export service. Your hard disk can be sent to AWS and it will be loaded into AWS cloud physically.

Storage gateway

Service connecting an on-premise VM with cloud based storage to provide seamless integration between on-premise data center and AWS.

4.0 Databases

RDS

Relational Database Service - MySql, Oracle, Postgresql, MariaDB

DynamoDB

Important subject for the exam. Amazon's NoSQL database service with push button functionality.

Elasticache

Caching your most popular queries in the cloud using in memory caching service - MemcacheD and Redis.

Redshift

Business Intelligence service. Data warehousing. Solutions architect exam - very important. Little for Dev exam.

DMS

Database Migration Service - Migrate from legacy based databases like Oracle to RDS using MySQL or other open source technology.

5.0 Analytics

EMR

Elastic Map Reduce - A way of processing big data.

Data Pipeline

Move data from one area or service to another. 

Elastic Search

Deploy, manage and scale Elastic search in AWS. Not covered in dev exam.

Kinesis

Platform for streaming data on AWS. Every time streaming is mentioned in the exam, it always refers to Kinesis.

Machine Learning

Use machine learning. Not required in the exam.

Quick Sight

Business Intelligency service. Like Cognos. Visualizations, ad-hoc analysis. Not in the exam.

6.0 Security and Identity

IAM

Control users, groups, roles, password rotation, dual authentication etc.

Directory services

Need to know different types of directory services. Needed for the exam.

Inspector

Allows to install agents on your EC2 instances and it inspects your EC2 instances to suggest security. Not required for Dev exam.

WAF

Web application firewall service. Not in the exam.

Cloud HSM

Hardware security module. Not in dev exam.

KMS

Key management service. Little bit for dev exam.

7.0 Management Tools

Cloud Watch

A monitoring tool for AWS environment.

Cloud Formation

Script your infrastructure. Important for DevOps exam.

Cloud Trail

Used for auditing to record changes made to your environment.

Opsworks

Configuration management service using Chef. Needed for Dev exam.

Config

Fully managed service for config history and change notifications. Not in dev exam.

Service Catalog

Create and manage a catalog of IT services. Not in dev exam.

Trusted Advisor

Automated service to scan your environment to suggest ways to save money and secure your environment. Needed for dev exam - what is it capable of and not capable of doing.

API Gateway

Allows developers to create, manage, maintain and secure APIs. Not covered now in dev exam, but can change.

AppStream

AWS's version of XenApp. Stream your windows applications from the cloud. Not on the exam.

CloudSearch

Manage and scale search solution. 34 different languages. Higlighting and geospatial searches. Not covered in exam.

Elastic Transcoder

Media transcoding service in the cloud. Converts from source format to different formats used for mobile, web etc. Needed for exam.

SES

Email service. Integrates with other AWS services. Needed for exam.

SQS

Decouples your infrastructure. Needed for AWS. First AWS service launched by AWS.

SWF

Simple Web Flow service. Developer can build and scale background jobs that have parallel or sequential steps.

8.0 Developer Tools

Not in exam yet.

CodeCommit

Private scalable git repos.

CodeDeploy

Automates deployments to any instance including EC2 instances or on-premise. Not in exam.

CodePipeline

Continuous Delivery service builds, deploys services and apps everytime there is a code change.

Read FAQs before going to exam. Not in exam currently.

9.0 Mobile Hub

Not in exam.

Mobile Hub

Build, test and monitor your mobile app.

Cognito

Allows to save mobile user data in AWS cloud. No backend coding. No need to manage infrastructure.

Device Farm

Testing your apps against real smart phones and tablets in the AWS cloud.

Mobile Analytics

Measure usage, revenue, new vs returning users etc.

SNS

Big topic in the exam. Amazon's simple notification service. 

10.0 Enterprise Applications

Workspaces

Virtual desktop in the cloud.

WorkDocs

Secure enterprise sharing service. Dropbox for the enterprise.

WorkMail

Amazon's answer to Exchange. Not in exam.

11.0 Internet of Things

Not in exam.


Wednesday, October 12, 2016

Trying @teamcity for #ContinuousIntegration - Part 3

In part 1, we looked at how to install and create a simple build job in Teamcity. In part 2, we looked at how to create an agent and run a build on the agent.

In part 3, we will look at how to build a Maven Java project. We already added the project to the build job in part 2, but we did not configure the build step to actually perform the compilation using Maven.

Auto detect build steps

It is as good as it sounds. I clicked 'Auto detect build steps' and the build step was automatically created to perform a 'maven clean test'.






















Check the checkbox in the table and click 'Use selected' to add the build step to your build job.

















Observe the 'mvn clean test' step in the table.











Run the build job

Run the build by clicking the 'Run' button at the top of the page. A live status and percentage of completion is shown on the project view.









Edit the maven goals

Click on the 'Edit' link on the maven build step row (second column from the right).








Change 'clean test' to 'clean test compile'.

















Run the build again using the Run button. The build log now shows the Maven build step.























The 'Maven Build Info' tab shows that the jar artifact was compiled successfully.
















I am curious to see more of this tool. The auto detect feature is very impressive.

Trying @teamcity for #ContinuousIntegration - Part 2

Continuing from where we got stuck in the prior post for running a build on Trying @teamcity for #ContinuousIntegration - Part 1. Before I started working on the agent, I deleted two of the three builds in the queue.

Create the Teamcity agent


I learned that I must have an agent running. I executed the following docker command to create an agent that automatically registered itself on the Teamcity server as an unauthorized agent.


docker run -d -e SERVER_URL="http://teamcity-server-instance:8111" \
--name=teamcity-agent \
-p 9090:9090 \
-v $HOME/docker-shares/teamcity/agent/conf:/data/teamcity_agent/conf \
-e AGENT_NAME=teamcity-agent \
--link teamcity-server-instance:TSI \
jetbrains/teamcity-agent














Authorize the agent

Click on the 'Unauthorized' link next to the agent and click 'Authorize' on the dialog  that pops up.














Navigate back to 'Connected' tab where only authorized agents are listed.














The build that was in the queue kicked off while I was trying to take a screenshot and I missed it on the screenshot. Navigate to 'Projects' and you'll see a 'Success' status on your build job.








More work needed

I was not expecting the build job to actually determine that this is a maven Java project and compile the code automatically. I looked at the build log and confirmed as much.
















The work we did above is truly just 'Hello World'. There is a build steps section that I didn't look for earlier. For now, I successfully connected an agent and kicked off a job. Moving on to build steps on the next post.





Trying @teamcity for #ContinuousIntegration - Part 1

Teamcity is a continuous integration software application from JetBrains. I am a Jenkins expert. I use it at work and I go deep into it with custom plugins and custom code to orchestrate builds and deployments. I use CircleCI and TravisCI for my github projects and these are just for fun. All these products are good in their own way. Jenkins is the mother of all CI tools. CircleCI and TravisCI are simpler to use and I really like the speed and agility of these tools. I would use Jenkins for enterprise and CircleCI/TravisCI for open source public domain projects.

I didn't realize that there is a free version of Teamcity until last night and the features are fairly decent. There is a 50% discount for startups and free for open source projects. This is welcoming.


Installation

Jumping right into it. I downloaded the windows version and the installation was fairly simple with the executable, but I could not get it to work. The browser would just show a blank page. I quickly switched over to my favorite way of doing things - Docker - and it worked right away.


docker run -d --name teamcity-server-instance \
-v $HOME/docker-shares/teamcity/server:/data/teamcity_server/datadir \
-v $HOME/docker-shares/teamcity/logs:/opt/teamcity/logs \
-p 8111:8111 \
jetbrains/teamcity-server



Open http://localhost:8111 and you see a very welcoming 'Teamcity First Start' page.


















I kept the installation simple. I chose the internal database option for the database and accepted the license agreement. I created a user account on the next form and I was brought to the settings page which shows off some pleasing features like:


  1. Version control integration
  2. Email, Jabber, IDE and Windows notifier
  3. Some popular plugin choices and tools on the right


















Hello Teamcity

Step 1: Click on 'Projects' link on the top left and click 'Create Project' - 'Manually'.
















Step 2: Enter project details and click 'Create'




Features observed: Very interesting feature titles that I will explore later.


  1. Build Configuration Templates
  2. Subprojects





















Step 3: Build Configuration


Click 'Create build configuration' - 'Manually'.
























I came this far to realize that I cannot do a simple 'echo "Hello Teamcity"'. I have to integrate with a version control system to proceed with this trial.



















Step 4: Integrating with Github


There are two tiny icons below the 'Repository URL' textbox for github or bitbucket. I chose github because I have a few projects there that I could build easily. Click on the 'Github' icon.













We have to register our local teamcity instance with GitHub. This was easier to do on Jenkins, CircleCI and TravisCI.






















Click 'register Teamcity' and create a new application token on GitHub. Just copy paste as stated in the screen above.





















When you click 'Register application', a new client id and client secret are created. Copy paste from github page to the teamcity github connection page and save. This takes you to a connection page instead of taking you back to your project configuration page. Just use the browser's back button to go back to your project configuration page. Now, the github icon will be showing as enabled.

Click on the github icon and sign in with your github credentials.












Step 5: Select project to build and trigger the build


I have a marstime github project. It is a maven project with java code. It is not perfect, not accurate and it is not well written. But, it compiles and runs.
















Select the github project and click create.

























Step 6: Run the build




Click on the dotted button next to 'Run'. Select 'run as a personal build' and 'clean all files in the checkout directory before the build' (Just some good CI habits). Then click on 'Run Build'.
















I was expecting teamcity to start building this code like Jenkins does on the master when it does not have agents. That didn't happen on teamcity. I clicked on 'Run' three times. Now there are three builds in the queue.














I'll have to create a teamcity agent docker container and hook it up to this teamcity server. Coming soon in part 2 of this blogpost.

Thursday, September 29, 2016

@Jenkinsci analytics with ELK @elastic on @Docker containers

This is a simplified way of setting up the integration discussed in Jenkins Analytics with Elasticsearch.

Step 1: Clone the project from github
git clone https://github.com/adityai/jenkinsElastic-Docker


Step 2: Build the Docker image for Jenkins with the logstash plugin.

docker build -t adityai/jenkinselastic-docker .


Step 3: Run the install.sh or execute the following docker commands

sudo docker run --name elasticsearch --hostname elasticsearch -p 9200:9200 -p 9300:9300 -d elasticsearch

sudo docker run -p 8080:8080 -p 50000:50000 -v $HOME/jenkins_home:/var/jenkins_home --link elasticsearch:es --name jenkins -d adityai/jenkinselastic-docker

sudo docker run -e ELASTICSEARCH_URL=http://elasticsearch:9200 -p 5601:5601 --link elasticsearch:es --name kibana -d kibana

Saturday, September 10, 2016

Python basics cheatsheet

Python Basics Cheatsheet




Python is an interpreted language.

Math, numbers and variables 

Mathematical Operators


Besides the usual operators (+, -, * and /), Python has exponent ( ** ) and negation ( - as a prefix to a number ).

Integers and Floats

Integers: 50
Floats: 3.1421

Order of operations

 PEMDAS: Parentheses, Exponent, Multiplication, Division, Addition and Subtraction

Using Variables

 No spaces in the variable name and must start with a character. Python style Pep 8 recommends lowercase words separated by underscore.

Rounding


import math
math.ceil(4.8) // will print 5

Other python libraries



Strings

Create strings

first_name = ‘Aditya’
last_name = ‘Inapurapu’

Concatenate strings

full_name = first_name + ‘ ‘ + last_name

Print strings

// Print one string
print(first_name)

// Print multiple strings one after the other with a space in between
print(first_name, last_name)

Python script

 Example: script.py
first_name = ‘Aditya’
last_name = ‘Inapurapu’
full_name = first_name + ‘ ‘ + last_name
print(first_name)
print(first_name, last_name)

Execute the following shell command to execute the python script:
python script.py

Comments

# is the comment character that makes the entire line in the python script as a comment.

Number to string

year = 1969
sentence = ‘The year was’ + str(year)
// A print statement automatically converts a number to a string
print(‘The year was’, year)

Single quotes in the string

famous_sketch = “Hell’s Grannies”

New line characters and tabs

famous_sketch1 = “\n\tHell’s Grannies”

String is an array of characters


greeting = ‘Hello World!’
#Prints H
print(greeting[0])
#Prints !
print(greeting[11])
#IndexError: string index out of range
print(greeting[12])

Built-in String functions

len()

greeting = ‘Hello World!’
print(len(greeting))

Sliceword = ‘Python’

# Prints ‘tho’
print(word[2:5])

Slice from the beginning of the string

word = ‘Good’
#Print from the beginning of the string to index 2
print(word[0:2])
#Also prints from the beginning of the string to index 2
print(word[:2])

Slice from the middle to the end of the string

word = ‘Good’
#Print from index 2 to the end of the string
print(word[2:])

Halfway index of the string using Integer division

// - is the Integer division operator which divides and always returns a whole number
word1 = ‘Good’
half1 = len(word1)//2
end1 = word1[half1:]
word2 = ‘Evening’
half2 = len(word2)//2
end2 = word2[half2:]
print(end1, end2)




Conditionals

< 
<=
==
>=
> 
!=

if num_knights < 3:
      print(‘Retreat!’)
      print(‘Raise the white flag!’)
else:
      print(‘Truce?’)
print (‘Knights of the Round Table!’)

if num_knights < 3:
      print(‘Retreat!’)
elif num_knights >= 10:
      print(‘Trojan Rabbit’)
else:
      print(‘Truce?’)

Input

day = input(“Enter the day of the week”)
print(‘You entered:’, day)

Convert from string to int

num_knights = int(input(‘Enter the number of knights’))
print(‘You entered: ‘, num_knights)
if num_knights < 3 or day == ‘Monday’:
      print(‘Retreat!’)




Nested if

if enemy == ‘Killer bunny’:
      print(‘Holy Hand Grenade!’)
if num_knights >= 10 and day == ‘Wednesday’
      print(‘Trojan Rabbit!’)
else:
      print(‘Truce?’)

Core data structures

List

# An empty list
empty = []

# A list of numbers
nums = [1, 2, 3, 4]

# A list of strings
words = [‘word1’, ‘word2’, ‘word3’]

# A list of mixed items
anything = [10, ‘hello’, ‘ahoy’, ‘mate’, 123.45]
# Prints 10
print(anything[0])
# Prints ahoy
print(anything[2])
# Prints hello ahoy
print(anything[1:3])

# Append items to the list
anything.append(‘end’)

# Remove item by value
anything.remove(‘hello’)

# Remove item by index
del anything[0]

# Remove items by slice
del anything[1:3]


Dictionaries

Dictionaries are equivalent to maps in Java where the data structure is a heap of key value pairs.
slang = {‘cheerio’: ‘goodbye’, ‘knackered’: ‘tired’, ‘yonks’: ‘ages’}
# Prints goodbye
print(slang[‘cheerio’])

# Empty dictionary
slang = {}

# Add key value pairs or update values
slang[‘knackered’] = ‘tired’
slang[‘smashing’] = ‘terrific’

# Delete items from the dictionary
del slang[‘cheerio’]

# If the key does not exist, a KeyError occurs and stops program execution. Use the get method to avoid this issue. If the key does not exist, ‘None’ will be returned.
result = slang.get(‘bloody’)
# if result exists
if result:
      print(result)
else:
      print(‘Key does not exist’)


Comparing Lists


my_list = [1, 2, 3, 4]
your_list = [4, 3, 2, 1]
his_list = [1, 2, 3, 4]
# Not equals returns false
print(my_list == your_list)
# Equals returns true
print(my_list == his_list)




Comparing Dictionaries

my_dictionary = {1:1, 2:2, 3:3, 4:4}
your_dictionary = {4:4, 3:3, 2:2, 1:1}
# Equals returns true
print(my_dictionary == your_dictionary)

List of lists

menus = [[ ‘Spam n Eggs’, ‘Spam n Jam’, ‘Span n Ham’],
      [‘SLT (Spam-Lettuce-Tomato)’, ‘PB&S (PB&Spam)’],
      [‘Spalad’, ‘Spamghetti’, ‘Spam noodle soup’]]
print(menus[0][1])

Dictionary of lists

menus = {‘Breakfast’: [ ‘Spam n Eggs’, ‘Spam n Jam’, ‘Span n Ham’],
      ‘Lunch’: [‘SLT (Spam-Lettuce-Tomato)’, ‘PB&S (PB&Spam)’],
      ‘Dinner’: [‘Spalad’, ‘Spamghetti’, ‘Spam noodle soup’]}
print(‘Breakfast Menu:\t’, menus[‘Breakfast’])
print(‘Lunch Menu:\t’, menus[‘Lunch’])

Loops

For loop

prices = [2.50, 3.50, 4.50]
# Print all prices using a loop
for price in prices:
      print(‘Price is’, price)
total = 0
for price in prices:
      print(‘Price is’, price)
      total = total + price
      print(‘total is’, total)
average = total/len(prices)
print(‘Average is’, average)

Generate a random number

import random
# Generates a random number from [0.0, 1.0]
r1 = random.random()
print(r1)
# Generates a random choice from a list
r2 = random.choice([1, 2, 3, 4, 5])
print(r2)
# Generates a random number between 1 and 1000
r3 = random.randint(1, 1000)
print(r3)

Simple range function


# Creates a list containing numbers 0 to 9
range(10)

for i in range(10):
      ticket = random.randint(1, 1000)
      print(ticket)

Start, Stop and Step range function


# Creates a list containing numbers ranging from 2005 to 2016 incrementing by 2
for i in range(2005, 2016, 2):
      print(i)
# The above prints 2005, 2007, 2009, 2011, 2013, 2015




More loops

# Append ‘spam’ to each word inside a list.
slang = [‘Knackered’, ‘Pip pip’, ‘Squidgy’, ‘Smashing’]
for word in slang:
      menu.append(word + ‘ Spam)
print(menu)
# Assign prices to each menu item in a dictionary.
menu = [‘Knackered Spam’, ‘Pip pip Spam’, ‘Squidgy Spam’, ‘Smashing Spam’]
menu_price = {}
price = 0.50
for item in menu
      menu_prices[item] = price
      price = price + 1
print(menu_prices)

# Print the keys in the dictionary
for item in menu_prices:
      print(item)

# Print key value pairs
for name, price in menu_prices.items():
      print(name, ‘: $’, price)

# Print without extra spaces
print(name, ‘: $’, price, sep=’’)

# Round/Format to 2 decimal places
print(name, ‘: $’, format(price, ‘.2f’), sep=’’)




While loop

x = 1
while x != 3:
      print(‘x is’, x)
      x = x + 1

orders = []
order = input(“What would like to order? (Q for Quit)”)
while (order.upper() != ‘Q’)
      # Find the order and add it to the list if it exists
found = menu.get(order)
if found:
orders.append(order)
else:
print(“Menu item doesn’t exist”)

# Check if the customer wants to order anything else
order = input(“Anything else? (Q to Quit)”)
print(prders)

Break

while (True)
      if order.upper() == ‘Q’:
           break
      else:
           …

Continue

while (True)
      if order.upper() == ‘Q’:
           break
      if order == ‘Cheeky Spam’:
           print(“Sorry, we’re all out of that!”)
           continue



Functions


Define a function

def average(prices):
      …
return avg

Call a function


numbers = [1, 2, 3, 4, 5]
my_average = average(numbers)
print(my_average)


main()

Good practice to have a function name main.

def average(numbers):
      total = 0
      for num in list:
           total = total + num
      avg = total/len(numbers)
      return avg

def main():
prices = [29, 21, 55, 10]
result = average(prices)
print(result)
main()




Files


Writing files


#Open the file in ‘w’ write mode
sales_log = open(‘spam_orders.txt’, ‘w’)
# If the file does not exist, python will create it for us.
sales_log.write(‘The Spam Van’)
sales_log.write(‘Sales log’)
sales_log.close()

# Append  mode
sales_log = open(‘spam_orders.txt’, ‘a’)

Reading files

# Read and print the entire file
dollar_spam = open(‘dollar_menu.txt’, ‘r’)
print(dollar_spam.read())
dollar_spam.close()

# Read one line at a time
dollar_spam = open(‘dollar_menu.txt’, ‘r’)
print(‘1st line:’, dollar_spam.readline())
print(‘2nd line:’, dollar_spam.readline())
dollar_spam.close()

# Read one line at a time in a loop till end of the file
dollar_spam = open(‘dollar_menu.txt’, ‘r’)
for line in dollar_spam:
        print(line)
dollar_spam.close()

# Read one line at a time into a list
dollar_spam = open(‘dollar_menu.txt’, ‘r’)
dollar_menu = []
for line in dollar_spam:
        dollar_menu.append(line)
print(dollar_menu)
dollar_spam.close()

#Remove the leading and trailing new line character
dollar_spam = open(‘dollar_menu.txt’, ‘r’)
dollar_menu = []
for line in dollar_spam:
        line = line.strip()
        dollar_menu.append(line)
print(dollar_menu)
dollar_spam.close()

Exceptions

Python has 60 plus exceptions.

Try, Except

try:
      file = open(‘sales.txt’, ‘r’)
      print(file.read())
except:
      print(“File doesn’t exist”)

Capture the error as a variable


try:
price = float(price)
print(‘Price =’, price)
except ValueError as err:
      print(err)






Modules


Installing modules


pip install requests

http://go.codeschool.com/pypi - for all available Python packages

JSON to format and share data


first_item = {‘name’: ‘Spam n Eggs’,
           ‘description’: ‘Two eggs with Spam’,
           ‘price’: 2.50}
second_item = {‘name’:’Span n Jam’,
‘description’:’Biscuit with Jam with Spam’,
‘price’: 3.50}
menu_items = [first_item, second_item]
print(menu_items[0][‘name], menu_items[0][‘price’], menu_items[0][‘desc’]

# Get JSON from a http request
import requests

my_request = requests.get(‘http://go.codeschool.com/spamvanmenu’)
menu_list = my_request.json()
print(“Today’s Menu:”)
for item in menu_list:
      print(item[‘name’], item[‘desc’], item[‘price’]




Create your own module



# orders.py
def print_menu(menu):

def get_order(menu):

def total_bill(orders, menu):


# spam_van.py
import orders

def main():
      menu = {‘Cheerio Spam”: 0.50, …}
      orders.print_menu(menu)
      orders = orders.get_order(menu)
      total = orders.bill_total(orders, menu)
      print(‘You ordered:’, order, ‘Total:’ total)
main()


Saturday, August 20, 2016

Django cheatsheet @djangoproject

Creating a Django project


django-admin startproject <project_name>

Create an app in a project


python manage.py startapp <app_name>

Run the project on a server


python manage.py runserver

Apply migrations on your app


python manage.py migrate

Import modules in your app


In <project_name>/<app_name>/views.py :

By default, the following views.py will be in your app folder:
from django.shortcuts import render
# Create your views here.

Add the http module in views.py as shown below:
from django.shortcuts import render
from django.http import HttpResponse

# Create your views here.
def index(request):
return HttpResponse('<h1>Hello Explorers!</h1>')

Map the view to a url


The default urls.py file in the <project_name> folder is as follows.

from django.conf.urls import url
from django.contrib import admin
urlpatterns = [
    url(r'^admin/', admin.site.urls),
]

Add a url to your app in the urlpatterns array. The following example is for adding a url ‘index’.

from django.conf.urls import url
from django.contrib import admin
from <app_name> import views
urlpatterns = [
    url(r'^admin/', admin.site.urls),
    # localhost/index
    url(r'^index/', views.index),
]




Refactoring the Project’s URLs Dispatcher


Replace ‘/index’ with ‘/’:

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    # localhost/index
    #url(r'^index/', views.index),
    url(r'^', views.index),
]

Separate project URL dispatcher and app URL dispatcher


Project urls.py

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url(r'^',
    include('main_app.urls')),
]

App urls.py


from django.conf.urls import url
from . import views
urlpatterns = [
    url(r'^$', views.index),
]




Templates


Register your App in Settings


In <ProjectName>/settings.py add your app in ‘INSTALLED_APPS’ section. In the example below, we are adding an an App named ‘main_app’ to settings.py.
The apps listed after main_app are default apps installed by Django.

INSTALLED_APPS = [
    'main_app',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

Create the template


In the <ProjectName>/<AppName> folder create a ‘templates’ folder and an index.html file like the one below.
<!DOCTYPE html>
<html>
<head>
     <title>MainApp</title>
</head>
<body>
     <h1>MainApp</h1>
</body>
</html>

In the <ProjectName>/<AppName>/ folder, edit the views.py file to render the index.html file instead of returning an HttpResponse.
from django.shortcuts import render

# Create your views here.
def index(request):
     return render(request, 'index.html')

Get Dynamic data in the template


Passing Dynamic Data to the Template


In the <ProjectName>/<AppName>/ folder, edit the views.py file to create new variables, context (json object) and render the html template with the context.

from django.shortcuts import render

def index(request):
name = 'Gold Nugget'
value = 1000.00
context = {
'treasure_name': name,
'treasure_value': value}
     return render(request, 'index.html', context)

Access the variable in the template from the context


In the <ProjectName>/<AppName>/templates/index.html template file, access and display the context variables using the django template language using the double curly braces.

<!DOCTYPE html>
<html>
<head>
     <title>MainApp</title>
</head>
<body>
     <h1>MainApp</h1>
     <p>{{ treasure_name }}</p>
     <p>{{ treasure_value }}</p>
</body>
</html>



Multiple dynamic data as list of objects


Create the list of objects as shown below using a class and an array list of objects of that class.

from django.shortcuts import render
def index(request):
     return render(request, 'index.html', {'treasures': treasures})

class Treasure:
     def __init__(self, name, value, material, location):
           self.name = name
           self.value = value
           self.material = material
           self.location = location
          
treasures = {
     Treasure('Gold Nugget', 500.00, 'gold', "Curly's Creek, CA")
     Treasure("Fool's Gold", 0, 'pyrite', "Fool's Falls, CO")
     Treasure('Coffee Can', 20.00, 'tin', 'Acme, CA')
}

Displaying an array list using the html template


Edit the index.html template to display the array list values.

<!DOCTYPE html>
<html>
<head>
     <title>MainApp</title>
</head>
<body>
     <h1>MainApaap</h1>
     <p>{{ treasure_name }}</p>
     <p>{{ treasure_value }}</p>
           {% for treasure in treasures %}
                <p>{{ treasure.name }}</p>
                <p>{{ treasure.value }}</p>
           {% endfor %}
</body>
</html>

Add a Django conditional


Add a conditional logic like the following example if required.

           {% for treasure in treasures %}
                <p>{{ treasure.name }}</p>
                {% if treasure.value > 0 %}
                      <p>{{ treasure.value }}</p>
                {% else %}
                      <p>Unknown</p>
                {% endif %}
           {% endfor %}

Styling templates


Django looks for stylesheets in a folder named static. Create a ‘static’ folder in <ProjectName>/<AppName> folder and add a style.css file.


In the html template, add the following Django template language line at the top of the file.
{% load staticfiles %}

Link to the style sheet by adding the following href to the link tag.
<link rel=”stylesheet” type=”text/css” href=”{% static ‘style.css’ %}”/>

Create an images directory in <ProjectName>/<AppName>/<static> folder. Add static images in this folder and use them in the html template.
<nav class="navbar navbar-default navbar-static-top text-center">
<a href="/"><img src="{% static ‘images/logo.png’ %}" alt="TreasureGram"></a>
</nav>

Models

A model is a mapping to a database table. It is about how we want to store data and retrieve data from the database.

Create a model


In <ProjectName>/<AppName>/models.py create a model like the one below.

from django.db import models
class Treasure(modesl.Model):
          name = models.CharField(max_length=100)
          value = models.DecimalField(max_digits=10, decimal_places=2)
          material = models.CharField(max_length=100)
          location = models.CharField(max_length=100)
          img_url = models.CharField(max_length=100)

 




Display the data from the model in the view


Import the Model into views.py of the <AppName> located in <ProjectName>/<AppName>.
from .models import <ModelName>

Create an object array of the Model and pass the object array as a parameter to the template in the index definition of the view.

def index(request):
     treasures = <ModelName>.objects.all()
     return render(request, 'index.html', {'treasures': treasures})

Model Field Types


Django
Python
SQL
models.CharField()
string
VARCHAR
models.IntegerField()
int
INTEGER
models.FloatField()
float
FLOAT


Django ORM


The Django ORM translates python code to SQL commands.

Migrations


A Migration is a step that needs to be performed whenever the model is created or updated. This allows the ORM to perform translations correctly.

Make a migration file


python manage.py makemigrations

Preview the SQL in the migration


python manage.py sqlmigrate <AppName> 0001

Apply the migration to the database


python manage.py migrate

Check if migrations need to be applied

Execute either the makemigrations or the migrate commands. If there are no migrations to be applied, the output of the commands will state that clearly.

Django interactive shell

Execute the following command to open the Django shell.

python manage.py shell

To play with App objects

Import the model from the app in the shell.

from <AppName>.models import <ModelName>

Execute one or more queries


QuerySet
Function
Equivalent SQL
<ModelName>.objects.all()
Display all objects in the model.
SELECT * FROM <ModelName>
<ModelName>.objects.filter(location = ‘Orlando, FL’)
Display all objects in the model where location is ‘Orlando, FL’
SELECT * FROM <ModelName> WHERE location = ‘Orlando, FL’
<ModelName>.objects.get(pk = 1)
Display the object where the value of the primary key is 1
SELECT * FROM <ModelName> WHERE <PrimaryKey> = 1


Create an object in shell


t = <ModelName>(name=’coffee’, value=’2.00’)
t.save()

Define and display a description for a Model


In models.py add the following to the class.

def __str__(self):
          return self.name

Now, whenever the object is queried in the shell, the output will display the actual value of the object.

Built-in Django Admin


The admin site is accessible from http://<hostname>:8000/admin.

Create super user

To use the admin site, a super user needs to be created. Execute the following command, enter your username, email and password when prompted.

python manage.py createsuperuser

By default, the groups and users are visible on the site, but not the models. The models need to be registered by the admin for them to show up on the site.




Register models with the admin

Register the model in <ProjectName>/<AppName>/admin.py to display the model on the admin site.

from django.contrib import admin
# Import the model
from .models import Treasure

# Register your models here.
admin.site.register(Treasure)