Sunday, August 6, 2017

Microservices 101, Docker & spring boot sample [Windows 10, home]

This is a very simplistic article :) if you are looking for a deep dive in microservices, see my state-of-the-art microservices full archeticture here: http://dev.basharallabadi.com/2019/03/part-1-spring-state-of-art.html

 

What are micro services?

It's an architectural model for web services that basically requires each service** to be completely independent and loosely coupled from other consumer services, or services that it depends on. and it's not a new idea but it's catching pace in today large scale web applications.


**(by service we mean a component that controls and implements the business logic in a self contained manner, like orders service, products catalog service, accounts management service, all of these have their domain and can be clearly separated) 

 Why micro services emerged ?


1- Easy to scale services: if you have a single application and all the services share the same code base and war (package) then if you receive high demand on one of the services, you will need to deploy more instances of the whole application but if each service was independent and has it's own code base and package then it will be easier to scale that specific service

 2- independent services: each service is managed and owned independently, it has its own lifecycle and upgrades, it can be written in any technology stack and it's self contained an encapsulated.

3- faster development cycles: each service is managed and owned independently from top to bottom, that means each team and each service based on its complexity are only limited by themselves which should minimize any noise that can slow them down in getting releases out.


Note:
you may wonder how can you do a complete transaction without the billing service talk with the order service for example.. the answer is that you will have your business transaction application above all these services and it handles communicating with each service and handles the orchestration among them.


What are the implications and technology stack when using micro-services?


1- The need for light weight hosts (application servers) due to the fact that each service will have its own application context, spring boot provides embedded web containers that fulfills this need.

2- usually you need to keep the documentation up to date of your service because other teams depend on it.

3- A lot of people now use containerized deployment using Docker.

4- for complex systems, a service registration server is needed to allow for run time changes without affecting consumers.(Netflix open source library for this is called Eurka)

Sample Project using Springboot, Docker & gradle:

I'm using Windows 10 home edition I'll be using intellj and git for code.

1- installing Docker: 

since I'm using windows 10 home edition, I had to install Docker toolbox, from here: https://www.docker.com/products/docker-toolbox

choose where to install it, and continue, it will install Oracle Virtual box and Git if not already installed, we will run docker in the 4th step.

2-creating the micro service project[ source code link at the end ]

 a- create the project directory tree and the empty build.gradle :
 -microservices-101
--src
---/main
----/java
----/resources
--build.gradle
--DockerFile

b-populate build.gradle with initial configurations:

first part states the build tools dependencies on spring boot and gradle docker
2nd part for the plugins
3rd part for the main class
4th part for the artifact meta data,
and the docker task part mentions docker file that will be used by docker plugin and prepares for the docker image creation by copying our jar to the docker staging directory in the build output folder of gradle

 the rest is just for maven dependencies.

buildscript {
 ext {
  springBootVersion     = "1.5.2.RELEASE"
  gradleDockerVersion   = "1.2"
 }
 repositories {
  mavenCentral()
 }
 dependencies {
  classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")
  classpath("se.transmode.gradle:gradle-docker:${gradleDockerVersion}")
 }
}

apply plugin: "java"
apply plugin: "application"
apply plugin: 'docker'
apply plugin: "org.springframework.boot"

compileJava {
    sourceCompatibility = 1.8
    targetCompatibility = 1.8
    mainClassName       = "com.bashar.microservice101.AppEntry"
}

jar {
    baseName = "api-cats"
    group    = "com.bashar"
    version  = "0.0.1-SNAPSHOT"
    manifest { attributes "Main-Class": "com.bashar.microservice101.AppEntry" }
}
task buildDocker(type: Docker, dependsOn: build) {
   applicationName = jar.baseName
   dockerfile = file('Dockerfile')
   doFirst {
      copy {
         from jar
         into "${stageDir}/target"      }
   }
}
repositories { mavenCentral() } dependencies { compile("org.springframework.boot:spring-boot-starter-web") }


c- DockerFile

This file contains the information the docker plugin will use to build our env into an image
from the docker hub:

FROM openjdk:8-jdk-alpine
VOLUME /tmp
ADD target/api-cats-0.0.1-SNAPSHOT.jar app.jar
ENV JAVA_OPTS=""
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar" ]
[Reference for step 2,3 : https://spring.io/guides/gs/spring-boot-docker/]


4- start docker:


open git-bash as administrator and navigate to Docker ToolBox dir, and run the start.sh there, it may take some time in the first time :






now with our docker server running we are ready to build our jar and put it in an image and run it.

1- first test that your application runs without docker 


gradle build && java -jar build/libs/<jar-name>, in my case (based on my gradle.build) it is api-cats-0.0.1-SNAPSHOT.jar

visit localhost:8080/cats/say
and you should get a "mewo"

that's good, now stop your application.

2- now run this gradle command to build your container:

gradle buildDocker

this will execute the gradle task we defined in our gradle.build, the logs that came out like this:
Sending build context to Docker daemon  28.66MB
Step 1/6 : FROM openjdk:8-jdk-alpine
 ---> 478bf389b75b
Step 2/6 : VOLUME /tmp
 ---> Running in 79d65abfecb2
 ---> 3fe9c14864cc
Removing intermediate container 79d65abfecb2
Step 3/6 : ADD target/api-cats-0.0.1-SNAPSHOT.jar app.jar
 ---> 91ca149552d7
Removing intermediate container 0fe80fd91ecc
Step 4/6 : ENV JAVA_OPTS ""
 ---> Running in bacda88cc38f
 ---> f2f3206d33c3
Removing intermediate container bacda88cc38f
Step 5/6 : ENTRYPOINT sh -c java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar
 ---> Running in 6c603b2ffcb8
 ---> 737ccde28f04
Removing intermediate container 6c603b2ffcb8
Step 6/6 : MAINTAINER bashar allabadi "basharlabadi@gmail.com"
 ---> Running in e693d744c5a3
 ---> afed0947a59b
Removing intermediate container e693d744c5a3
Successfully built afed0947a59b
Successfully tagged api-cats:latest
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.


and that task reads the docker file, downloads the image from Docker hub, mounts /tmp for our application (see the reference link on why they did that, it's needed if spring boot app to write files)
and then docker copied our jar from target/<jar-name> to app.jar, set env variable JAVA_OPTS to empty, and set the entry point of this image to run our app.jar.

note: the java.security.egd is explained in the reference link, it's optional and irrelevant to docker.


now if you run

$ docker images

you will see:

REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
api-cats             latest              afed0947a59b        26 minutes ago      115MB
openjdk              8-jdk-alpine        478bf389b75b        5 weeks ago         101MB


this is cool, that means our image is ready to be used

3- containerize the image and run it :


$ docker run -p 8080:8080 -t api-cats
(the -p is to do port mapping in case you want to change it, or you have multiple instances of it)




NOTE: that you can't have two images containers running on the same port, so you have to stop it first

$ docker ps

output:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
ca8795df86f4        api-cats            "sh -c 'java $JAVA..."   21 minutes ago      Up 21 minutes       0.0.0.0:8080->8080/tcp   peaceful_boyd


$ docker stop <container_id>


to access the app in this container, and because we are on windows, you have to go through the virtual box first that is hosting the Docker container engine, the IP address is in the first picture above


Note:
there are alot of examples like this on the web, the problem that mainly faced me was that I'm using a windows machine, and I couldn't use the docker plugin without docker file the way this tutorial is doing: https://medium.com/@hudsonmendes/docker-spring-boot-microservice-with-gradle-9785087e7992
however, it all leads to the same results, so just be patient if things don't work, take a step back and you maybe have to do a work around until you deeply understand how things glue together.


now with that is done, let's emphasize the value of what we did to the Micro-services approach, by:

1- our micro service is up in a container and isolated from the host machine OS changes
2- our service container can be pushed to Docker hub for anyone to use
3- we can create other instances of the same image and run it so we can scale very quickly


we did #1, lets do #2 & #3:

Push and share our micro-service image

a- create repository on docker hub if we don't have

  https://hub.docker.com

b- login through the shell 


$ docker login --username=basharlabadi --password=<password>

it should say login succeeded

c-now let's give our current microservice image a version (label, tag) 


 so that we can identify other versions in the future:

run:

1- $ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
api-cats             latest              afed0947a59b        About an hour ago   115MB

2- $ docker tag afed0947a59b basharlabadi/microservice-101:v1

3- $ docker images
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
basharlabadi/microservice-101   v1                  afed0947a59b        About an hour ago   115MB
api-cats                        latest              afed0947a59b        About an hour ago   115MB


notice the new image now in our images list.

d- push the image to our hub repo:

$ docker push basharlabadi/microservice-101:v1
The push refers to a repository [docker.io/basharlabadi/microservice-101]
73288e022ebd: Preparing
e378420761b2: Preparing
b0f743408169: Preparing
5bef08742407: Preparing
b0f743408169: Mounted from library/openjdk
e378420761b2: Mounted from library/openjdk
5bef08742407: Mounted from library/openjdk
73288e022ebd: Pushed
v1: digest: sha256:bf247204a436f98d2a7d9ac38418d9393978871765fad2e9bd048cdc4211dd5f size: 1159


Testing our pushed image 


1- remove all images from our docker engine 


using this command:
docker rmi -f <image_name>


2- when we are done with that, we will pull our image from docker hub:


$ docker pull basharlabadi/microservice-101:v1
v1: Pulling from basharlabadi/microservice-101
Digest: sha256:bf247204a436f98d2a7d9ac38418d9393978871765fad2e9bd048cdc4211dd5f
Status: Downloaded newer image for basharlabadi/microservice-101:v1

Bashar@DESKTOP-7I24OAD MINGW64 /c/dev/blogging/micro-services-101


$ docker images
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
basharlabadi/microservice-101   v1                  afed0947a59b        About an hour ago   115MB


cool our image is back again, from the hub not locally let's try and run it



anyone basically with Docker can now run our microservice without jdk, spring, or anything, it's crazy :)



- note the ports are different than the previous time


Scaling up our service with Docker


let's now run another instance of this service to handle the demand on our cats meows:

using the same command but different port we can roll out another instance, while the other is up and running normally:

$ docker run -p 8898:8080 -t basharlabadi/microservice-101:v1

and to see the instance that we have running :

$ docker ps



as you can see the two instances are there and there is a third instance that we started before and I forgot to stop.



full source code here:


Thanks

Thursday, July 27, 2017

Algorithms review

looking back at algorithms and data structures which are the backbone of programming..

here is a Merge sort I wrote using JS fiddle:


Sunday, January 8, 2017

Spring security OAuth2 + spring boot, Part 2

In this 2nd part I made a more complex example to be more practical in real life, and created a web app that is gets its users from db and stores the clients, and tokens in db instead of memory so it's more scalable.

features:

1- users & roles in database
2- oauth related info stored in db (tokens, clients, approvals, etc..)
3- two APIs : a- cats apis, b- dogs api; each one of these can be accessed only if the access token has a proper scope (cats scope and dogs scope)


the app uses h2 in memory database so that it be self contained and no external setup is required

and here is the source code :

https://github.com/blabadi/oauth2lw-parent/tree/master/oauth2lw-client-jdbc


show case:

1- get access token to with scope cats:

a- get the auth code :

go to : localhost:8080/oauth/authorize?response_type=code&client_id=cats-client&redirect_uri=http://localhostcats/redirect

2- login with a user
3- approve the scopes :



4- get the authorization code


5- exchange the code for a token



6- access cats api:


7- try the dogs api


try to get a valid token to access the dogs api by yourself should be similar to the cats process..


get familiar with the source code, it's only a step up and we only used the classes from the oauth2lw-core we added the jdbc datasource and the token store and passed these to our configures.

Saturday, January 7, 2017

Spring Security OAuth2 + Spring Boot, minimalist working, extendable configuration - Part 1

Hi

been a while again..

today i'm back to write on how to write the simplest possible spring boot app protected with spring security oauth2, but i'll also make the project as a library jar that can be reused for any new projects to kick off faster.

 and it will be a simple ready to override functions that will let you focus on configuring only things that you need to worry about without having to worry about annotations or other spring security stuff.


Overview

Spring security oauth 2 has three main components :

a- Authorization server: to handle tokens/authroization codes and user approve/deny
b- Resource server: protects the actual apis that we want to be protected by oauth2
c- spring web security configs to manage users authentication

and as simple as it sounds to get these three in one simple hello world example required hours and hours of searching and debugging dont ask me why..


Pre Requests :
1- you know spring security & spring web security concepts and how to configure it like basic auth or form login


so without further due here are the three classes you will need to get oauth2 up and running :

1- Spring security:

https://github.com/blabadi/oauth2lw-parent/blob/master/oauth2lw-core/src/main/java/com/bashar/oauth2lw/core/SecurityConfig.java

this class extends the Spring web security configurer to allow us to set how we want to protect our authorization server, and to set up our users (Resource owners) store. here it's in memory but in your application you will need jdbc so you can just extend and override this.
by default this protects /oauth/authroize api with the default spring web security login form

when you need to override that form you should override this method :

protected void configure(HttpSecurity http)

and set the url of the login page of your choice using the usual spring web security configs.

2- Authroization server:

https://github.com/blabadi/oauth2lw-parent/blob/master/oauth2lw-core/src/main/java/com/bashar/oauth2lw/core/AuthorizationServer.java

here this configuration server has two main parts :

A-  configure(ClientDetailsServiceConfigurer clients)
this method configures the clients of our oauth2 resource, and clients in 90% of the cases in the world of oauth2 represents the 3rd party apps that want to access the resource owner data on his behalf (as example of that is when you allow a web or mobile application to access your instagram photos or facebook friends list)

B- configure(AuthorizationServerEndpointsConfigurer endpoints)
this method let you configure the oauth2 endpoints settings as: where to store the tokens, codes, and many other customizations that are usually not needed (the most important one is setting the token store)

3- Resource Server.java

here we can configure 2 important things

https://github.com/blabadi/oauth2lw-parent/blob/master/oauth2lw-core/src/main/java/com/bashar/oauth2lw/core/ResourceServer.java


1- configure(ResourceServerSecurityConfigurer resources)
this can allow us to configure the id of this resource server (becomes handy if we have multiple resource servers and we have access rules for different clients on each of them) and the more important config is the token store, i.e. where to verify the tokens that we receive in the requests and make sure they are authentic and created by our Authorization server.

2- configure(HttpSecurity http)
this is important and allows us to configure the urls that we want to protect under the oauth2 protocl in this resource, anything we specify here will require a valid token that can access this resource propery and in my default configuration i protected any api under the path /protected to be fully authenticated, now this is app specific configs because each app has its own apis and roles and security rules, so make sure you understand how to use the HttpSecurity builder

it worths mentioning the oauth2 spring libraries provide some exprissions like hasScope() and other security functions related to oauth2 security.


now how to use these classes?

you just have to add this jar in your deps in maven or gradle or whatever and import these 3 classes to your spring configurations, I have made an example in this client project:


https://github.com/blabadi/oauth2lw-parent/blob/master/oauth2lw-client/src/main/java/com/bashar/oauth2lw/client/Oauth2lwClientApplication.java

this is simple spring boot application and in it, just for practicing i overrode the method that configures clients in Authorization server and added my own client.. this can be converted to jdbc or whatever you want

and for the other 2 classes I haven't changed anything so I just imported them as is, if I haven't changed anything in Authorization server I would have also just imported it, but it will be scanned automatically since we extended it and annotated it with @Configration


and when we run the application and as for a protected (although non existing) url like :



so to get a token there are many flows but the more complex and more typical that we would use oauth2 for is authorization code flow, so to obtaina  token go to the browser and type in :

http://localhost:8080/oauth/authorize?response_type=code&client_id=trusted1&redirect_uri=http://localhost:9090/redirect

this will redirect us to login page where this is similar to when you are redirected to facebook by a third party app to login and approve its permissions to get some of your data

so we login with our dummy in memory user setup in the spring security configs:


then you get the approval of scopes page


and if the user approves, then the server will issue an authorization code and return it to the 3rd party app on the redirect uri provided 


the 3rd party server now should exchange that code with our server to an access token




after getting the token it can now use it to access the protected resoruce that we couldn't access before




Simple and clean and ready to be used out of the box just get the jar of the oauth2lw-core and drop it in your project and you will have oauth2 security ready.

note that the core lib pom already have the required deps:

org.springframework.cloud spring-cloud-starter-oauth2

so no need for you to worry about it..


Istio —simple fast way to start

istio archeticture (source istio.io) I would like to share with you a sample repo to start and help you continue your jou...