Basics of Cryptography

The wiki page on Cryptography gives you some basic ideas already.I will just try to elaborate a bit more to explain the topics at a very introductory level.

In the earliest days of Cryptography, it was all about Symmetric Cryptography wherein a private key was used to encrypt data at the sender’s end and decrypt data at the receiver’s end, so that confidentiality(one of the¬†CIA Triad) of the over-all system is maintained.But integrity(another CIA Triad) was missing at it’s core(availability, which is again another CIA Triad is more of an Application Specific stuff and related issues like Denial Of Service can be handled using techniques like Throttling).Also key(the private key) management was an issue.Imagine ways to securely sharing the private key with your client(s) – could be via emails, over the phone or you might be using a very faithful parrot/pigeon ūüôā or some other weird way – but is that a feasible way if it’s a very large system with huge number of senders and receivers? Nope, not at all.Key management of Symmetric Cryptography sucks for very large systems wherein there are hell lot of systems and hell lot of clients trying to communicate with those systems.

The limitations of the Symmetric Cryptography(as mentioned above) paved the way to Asymmetric Cryptography(aka Public Key Cryptography) wherein both public and private keys are used to share some data over the wire between the sender and the receiver.These keys are generated mathematically using Number Theory techniques based algorithms and are always used in pairs Рif the public key is used to encrypt the data then the private key is used to decrypt the data and vice versa.Also, most of the algorithms allows such a pair to be generated only once which is a globally unique pair.

But how does Asymmetric Cryptography take care of Confidentiality and Integrity.Well, let’s ¬†have a look at that –

  1. Confidentiality -> First of all,¬†we need to understand who should generate the keys to encrypt or decrypt the data¬†– the sender of data or the receiver of data? Well it turns out that in real world systems,¬†the receiver(e.g some Web Services Provider) generates the pair(both public and private keys) and sends the public key over the wire to the client(s) and then the client(s) use this public key to¬†encrypt the data and send the encrypted data back to the receiver which uses the earlier generated private key to decrypt the encrypted data.So if there is a (wo)man in the middle attack, (s)he wont be able to understand the data that is going over the wire since (s)he doesn’t have the private key to decrypt the data and that’s how confidentiality is achieved.But why is it not the other way around i.e. why not the keys are generated by the client(s)? Well, I am not going to go any deeper on this since this can be a good exercise for the reader. Just think for sometime and I am sure you will be able to get the answer.
  2. Integrity -> Whatever mentioned above regarding Confidentiality doesn’t let the (wo)man in the middle to understand the data but (s)he can tamper the data due to which even if the receiver is able to decrypt the data that won’t be the client(s)’ intended data to share with the receiver which is a major problem.So how does Asymmetric Cryptography handles this problem of Integrity? Well, say hello to Hashing.The client(s) separately hashes the data and encrypts the data and sends both the hash and the encrypted data over the wire which the receiver on receiving decrypts the data and hashes the decrypted data at its end and checks if the hash it has generated is the same that was sent by the client(s) and if it matches, everything is fine else it’s alarming and thus integrity of the over-all system is maintained. In-fact the Hashing approach could have been used for Symmetric Cryptography as well but the earliest techniques of Symmetric Cryptography never used that but in recent days most Symmetric Cryptography techniques uses this Hashing technique for integrity.

But is there still some flaw with whatever explained till now? What about the public key shared across the wire? Is there any loop hole there?Think like a hacker. Think before reading further.

Yes, there is a loop hole.What if the the hacker((wo)man in the middle) traps the public key (can be easily done since it’s publicly exposed to everyone over the wire) and replaces it with it’s own public key and so when it receives the encrypted data from the client(s), it will use its own private key(generated by the hacker while generating the public key) to decrypt the data and that’s a very serious problem.So in this case, the client(s) are not sure if they are using some genuine and valid public key which they can believe/depend on.So how do we tackle this problem? Dont worry :)! Digital Certificates are there to rescue you from this problem.

So how does a Digital Certificate ensure that the public key is not misused by some hacker and it’s a valid one? Let’s see how.

There are lots of CAs(Certificate Authorities)¬†(e.g. Verisign) which issues Digital Certificates to systems on request of the system owners(but after doing all sort of needed verification/cross-checking about the system and their owners).The system owners shares the public key(alongwith sharing some other security attributes used by the system) with the CA ¬†and then the CA uses this data to generate the Digital Certificate which is¬†a combination of encryption of this data(encrypted by using a private key) and a digital signature(which is created by hashing the public key shared by the system with the CA and then applying another private key encryption on the hash).Now such systems doesn’t send the public key directly over the wire to it’s client(s) but uses Digital Certificates to share the public key with the client(s).The client(s) communicates with the CA to get the public key to decrypt the Digital Certificate to get the actual public key to be used for encrypting the data to be sent to the systems/receivers.For more info on Digital Certificates refer Understanding Digital Certificates.

So now, I have another question.Where does https using SSL/TLS comes into picture amongst all these? To explain this I have another question – how does the systems communicate/negotiate with their clients about what algorithm is going to be used for encryption using public key(needed for Confidentiality as explained above) and which Hashing algorithm is going to be used(needed for Integrity as explained above)? That’s where SSL/TLS comes into the picture. SSL/TLS serves as the communication/negotiation medium between the systems and their clients as to which algorithms are to be used for confidentiality and integrity which are mandatory for the to and fro secure transmission of data between the systems and their clients.

Well that’s all for now.I hope I have been able to explain the Basics of Cryptography in a simple way so that it’s understandable even by an amateur/novice(just like me ūüôā ) in this field.There are some advanced topics like Certificate Chaining, Symmetric Encryption for Session Data, Symmetric Encryption for Streaming Data¬†etc which are beyond the scope of this introductory article on Cryptography(by the way, Symmetric Cryptography is not extinct but still has lot of applications).

N.B. -> For further study in Cryptography (& Information Security in general), refer the courses offered by Stanford University on Computer Security.


Kool Resources for GATE Electronics and Communication Engineering Examination

Well, this is a totally off the track article compared to the other software development articles¬†published in this¬† blog.¬† Although I am more passionate about Software Development NOW but “Once Upon a time in Mumbai” I had a quite a bit of passion for Electronics and Communication Engineering (being an Industrial Electronics Undergraduate).

I prepared and appeared for GATE(Electronics and Communication Engineering) twice – the first time I could not know my score (since at that time, which was back in 2004-2005, they didn’t give you the score if you scored less than 75 percentile) and the 2nd time I scored a fair percentile but that was not good enough to get into IITs or IISc Bangalore (at that time, you needed atleast a score of 95 percentile to get into IITs and atleast a score of 99 percentile for IISc Bangalore). Although, there were(& still are) other good universities in India but somehow I took the decision then that if I am going to do Graduate¬† studies then it’s gonna be from IITs or IISc Bangalore else “nope”. Also doing a job in a software firm then seemed to be more lucrative compared to studies(“Earning” had overridden “Learning” – could see $s in my eyes just like Richie Rich – but then again with time understood the realms of Earning and Learning w.r.t College life and Job life – in College life you have to pay for Learning while in Job life you get paid for Learning).

Based on my long back preparation and some research(whenever I get a bit nostalgic about GATE and Electronics & Communication Engineering in general and that happens atleast once in a year, I do some random googling), mentioned below are some suggestions/resources as per the current GATE syllabus(as of this writing) for Electronics and Communication Engineering

1) Engineering Mathematics – To understand the concepts, I doubt that there is anything better than Engineering Mathematics and Advanced Engineering Mathematics by Ken Stroud. For practice problems one can refer, Higher Engineering Mathematics by B.S.Grewal. I wish I knew about these resources when I appeared for GATE(atleast when I appeared for the 2nd time) since this section was one of the main reasons for not enough GATE score, even when I appeared for the 2nd time.

2) NetworksFundamentals of Electric Circuits, Engineering Circuit Analysis, Circuit Theory : Analysis and Synthesis, Introduction to Electric Circuits, Linear Circuit Analysis, Valkenburg’s Network Analysis and ¬†Introduction to Electrical Engineering are some good texts to understand the concepts. In-fact the last text mentioned touches almost all the breadths of Electrical, Electronics and Communication Engineering at a very introductory level providing some great real world examples.For problems, one can refer Allan’s Circuits Problems.

3) Electronic Devices – Although there are lots of resources in this area, but I think one should refer Principles of Semiconductor Devices and Fabrication Engineering at the Micro and Nano scale for this discipline.

4) Analog Circuits – Textbooks by Milman & Halkias and Taub & Schilling were(& still probably are) suggested in the university from which I undergraduated (& probably most of the other Indian universities suggest the same). But these textbooks did nothing more than confusing me more and more.I got enlightenment in this area after reading Microelectronics Circuits by Sedra Smith and Microelectronic Circuits and Devices by Mark N. Horenstein. Another good text (a bit advanced) in this area is CMOS Analog Circuit Design.

5) Digital Circuits –¬†Microelectronics Circuits by Sedra Smith is helpful here as well. Other good resources are Fundamentals of Logic Design(having some of the most intriguing problems in Digital Design),¬†Computer Arithmetic and¬† Microprocessor Architecture, Programming and Applications with the 8085. If you have interest in Microprocessors , Micro-controllers and the related area of Embedded Systems, you can refer¬† Micro-controllers and Microcomputers: Principles of Software and Hardware Engineering and Embedded Systems Courses by Jonathon W Valvano. Although not part of GATE syllabus as such but a related field to Digital Circuits(at a much high level) is Computer Architecture and Organization and if you have interest in this area, you can refer¬†Principles of Computer Hardware and these superb suggestions. And anyone interested in VHDL in general, can refer Digital Systems Design Using VHDL.

6) Signals and SystemsLinear Systems and Signals by Lathi, Signals and Systems by Chen and Signals and Systems by Oppenheim et al(mind it, this is not an easy text but it’s mandatory) are some good resources to go for this toughie field. Signal Processing and Linear Systems again by Lathi is another good text in this area.Other texts which can be helpful as well in this area are Design of Analog Filters by Valkenburg¬† et al and Digital Signal Processing again by Chen.

7) Control Systems – Although there are lots of resources for this complex topic as suggested here but personally I think that Control Systems Engineering by Nagrath and Gopal, Design of Feedback Control Systems by Stefani et al, Process Control and Instrumentation by Johnson, the book by Ogata, Modern Control Systems Theory again by Gopal and Systems and Control by Zak are the ones to be read to understand the concepts in-depth.

8) Communication Systems – Again you will get an ocean of texts related to this real toughie field but I feel that Fundamentals of Communications Systems by Fitz, Modern Digital and Analog Communication Systems by Lathi, Principles of Digital Communications by Gallager should be the ones to be referred. A Communication Systems course in general needs knowledge of Probability,Statistics and Random Processes and for that one can refer this online text.You can also refer the suggestions by a real Communication Systems Engineer for doing practical work in Communication Systems.

9) Electromagnetics – I still can’t believe that I passed this most complex subject of Electronics and Communication Engineering in the first go during my College days(I passed with just the pass marks). This was the most dreaded subject in our times(& probably still is). In-fact this was the most important reason for me not getting an enough good GATE score even in the 2nd attempt(the 2nd most important reason being Engineering Mathematics). As far as suggestions for this most complex field of Electronics and Communication Engineering is concerned, if you want, you can go via the route suggested by the Amazon Reviewer¬† but I would personally suggest to go for Elements of Electromagnetics by Sadiku, for it’s lucid style.I wish I had known about this text in my college days or during my GATE preparation but I came to know about this text quite later after doing some googling . And by any chance if you are interested in doing research in Electromagnetics, you should go through this paper.

N.B. -> Alongwith the above mentioned texts one should also have the Schaum’s Series books handy(get a hold of the ones relevant to your discipline), since Schaum’s Series books are the ones which not only will help you like anything before the university exams but before competitive exams as well. Also the E-Learning Courses from the IITs and IISc¬† can be very helpful. Also visit the relevant courses from US universities like MIT, Stanford, Harvard, UC Berkeley, Princeton University etc. And by the way, if you have a neck for Real World Practical Electronics stuffs in general, go for Practical Electronics for Inventors.

N.B. -> If one is looking for IIT JEE resources, then probably ASKIITians is¬† the best website to explore.Most suggestions for subjects for IIT JEE(atleast Maths and Physics) point you to some Russian origin book and to get a hold of most of the well known Russian books(translated to English language) visit Mir Titles. Again, Schaum’s Series texts are very helpful here as well.I did prepare for IIT JEE but did not appear for it for various weird reasons but that preparation helped me a lot in preparing for some courses in my College Freshmen and Sophomore days.

P.S. -> My only intent in writing this post(although totally out of track compared to the software development posts published in this blog) is to help interested candidates out there to get a good score in competitive exams like GATE or DRDO(by the way, DRDO seemed more tougher compared to GATE, atleast in my eyes – although all the questions were multiple choice questions but when I was trying to solve them, I felt like I was put into some combat to solve some research problem in most of the cases). And even if you are not able to crack any competitive exam, hopefully, the above mentioned resources (along-with some practical real world hands-on experience) will make you a better Electronics and Communication Engineer(which is more important) but that again is applicable if you really want to pursue a career in Electronics and Communication Engineering.

If anyone knows of any other good resource pertaining to GATE Electronics and Communication Engineering Exam (or any other Indian competitive Exam for the same discipline), then please leave a comment.

Rendezvous with Microsoft

Back in November 2014, I went through some round of interviews with Microsoft Hyderabad (MSIT). Sharing that experience with you all.

Microsoft has the following business units in India ‚Äď

  • Microsoft Corporation (India) Pvt. Ltd. ‚Äď It has offices in Bangalore, Ahmedabad, Gurgaon, Chennai, Hyderabad, Kochi, New Delhi, Pune, Mumbai, and Kolkata. Mainly deals with the Marketing/Sales of it‚Äôs products and services.
  • Microsoft IT (MSIT) ‚Äď It has only one office in Hyderabad. Deals with internal IT projects. Internal doesn’t mean only internal to MSIT but can cater to projects for customers of MS who are using MS offerings e.g. a website dealing with Windows Azure licensing.
  • Microsoft India Development Centre (MSIDC) ‚Äď It has two offices in India, Hyderabad and Bangalore. Deals with Development of some of the products offered by MS.
  • Microsoft Services Global Delivery (MSGD) ‚Äď It has two offices in India, Bangalore and Hyderabad. Deals with consultancy and providing solutions to different industrial sectors like Banking and Financial Services, Insurance, Retail, Health Care, Life Science, Manufacturing and Logistics etc.
  • Microsoft India Global Technical Support Centre (IGTSC) ‚Äď It has two offices in Bangalore. Deals with the Technical Customer Support for various products and services.
  • Microsoft Research India (MSR) ‚Äď It has only office in India, Bangalore. Deals with Research in almost all the areas of Computer Science and Engineering.

When I got the call from MSIT Hyderabad for an interview, for an opportunity to work with Microsoft, I was not sure what role was I supposed to play if I get selected(one true confession – I did not ask for the role since I wanted to go through the interview process to know about MS interviews and I did not want to miss my first opportunity to get drilled by one of the Software Giants and believe me, “the experience was quite different” – forget about “I got selected or not”).

They will mostly ask questions related to the skill-set and projects you have mentioned in your resume. There were 4 rounds of interview (each, of around 1 hour) – 2 telephonic and 2 face to face. Now, let me give you a brief about the sort of questions I was asked –

1) 1st round (telephonic) –

a) ASP. NET(Web Forms,MVC & RESTful Web API) –¬† Questions on ASP .NET pipeline, differences between ASP.NET Web Forms and ASP.NET MVC w.r.t the pipeline, In which scenarios one is better over the other. In ASP.NET Web API (or any Restful Service implementation), is it recommended to use PUT and Delete verbs? Why not? How would you then implement functionalities provided by these verbs? Also w.r.t REST, questions might be asked related to the Richardson Maturity Model.

N.B. -> Some more ASP.NET MVC questions

b) .NET Core Framework – Differences between Dispose and Finalize, How the Garbage Collector works, Brief about Garbage Collection Algorithms, Garbage Collection Generations etc.

c) Concurrent Programming – What is TPL(Task Parallel Library), Explain the Concurrent Programming Evolution in .NET and the rationale behind the design choices. Given a chance to redesign TPL, how would you go about it.

d) LINQ and Entity Framework – When to use IQueryable over IEnumerable and vice versa, Are the number of query operators in IEnumerable and IQueryable same?(yes or no and you need to justify your answer about why do you think so i.e yes or no). Impact of Deferred execution in IEnumerable and IQueryable, Best Practices for IQueryable and IEnumerable.Does Entity Framework handle Transactions by default and if not then how to handle Transactions while using EF, What is something in EF that you don’t like and how would you go about re-designing EF to mitigate the flaw?

N.B. -> Some more EF questions

e) Architecture РSome basic questions on Extensibility and Maintainability. Questions on Enterprise Application Architectural Patterns like Repository Pattern, Unit of Work Pattern etc. Questions on Domain Driven Design (including CQRS and Event Sourcing Рscenarios where these should be used). Relation between DDD Bounded Context and Entity Framework DBContext. What is Eventual Consistency and explain a scenario where Eventual Consistency should be the optimized way to go (providing proper rationale for the same)?

2) 2nd round (telephonic)-

a) WCF РSome scenario based  questions on WCF Instancing, Concurrency and Security(including certificates). Also some questions on WCF Architecture and Extensibility(again scenario based).

N.B. -> Some more WCF questions and WCF Security Interview Questions

b) Software Design РYou will be given a scenario (fictitious or non-fictitious) and you need to design that. Basically what Object Oriented Principles (including S.O.L.I.D principles) and Design Patterns you will use to design the scenario and you will have to give proper rationale for your design decisions (mainly the Design Pattern choices). Also you might be asked, why you opted for Design Pattern A rather than Design Pattern B for some given scenario.Also you might be asked questions on IoC/DI like have you ever used IoC to implement something  other then DI? If yes, why did you opt the IoC approach rather then some simple approach and how did you do that?

c) Performance – How do you analyze a performance issue end to end, Tools to find Performance bottlenecks (mainly what sort of Profilers you have used, although Fiddler also acts as a good tool to some extent to troubleshoot Performance related issues), questions on out-proc Session usages,Which one is better – Session State Server or SQL Server based Session Server and why. CDNs & JS Minifiers. ADO.NET and Entity Framework related Performance issues, Tackling DB Performance issues(indexing, sql hints, execution plan, hardware scaling etc),Even after applying proper indexing, sql hints, your DB is performing slow ‚Äď what could be the reason? Optimistic and Pessimistic Concurrency related questions and their relation to Performance as a whole.

3) 3rd round (face to face) –

a) Current Project Architecture – Explaining the architecture of the most recent project you have mentioned in your resume. Will ask you questions based on the architectural and design decisions taken and any flaw found somewhere, you need to give proper justification for the same. If both of you agree that there was a design flaw then need to give the solution that will refactor that design flaw.

b) Technology Selection in your Current Project – Why did you use LINQ in your project? In which scenarios LINQ should not be used? Why Entity Framework and not ADO.NET? In which scenarios you should use only Multi Threading and not Parallel Programming? When is it really beneficial to use Parallel Programming? Is there any advantage of using Parallel Programming in a single core machine? Why did you use MQs and why not ESBs or Brokers(& why not Microservices approach)?

N.B. -> Although during Nov 2014 I was part of a pure Web Application project(using some custom proprietary UI and App Frameworks, which was challenging) but was there for just around 3-4 months only, so I just did a cursory mention about this project in my resume but strongly mentioned about the project that I had worked before the Web Application project extensively(for almost 2 and a half years), which was a Workflow and Integration based project(although this project also had UIs, but was developed using normal ASP.NET MVC, I feel the challenging part in this project was the Long Running Processes and Integration with other Java and Mainframe based applications via MQs). In the Workflow and Integration project, the technologies mentioned in the aforementioned questions (section 3b) were used and that’s why I was asked these questions (section 3b). As I already mentioned earlier as well, they mostly ask you questions related to whatever you have mentioned in your resume.

c) Performance – Questions asked were almost similar to questions asked in interview round 2(c). Also another question asked was – What is the significance of SQL Server Stats?

d) Process, Project Delivery and Team Handling – What is so great about Agile and TDD? In what kind of Projects would you opt for them? How to handle changing requirements? Questions on Automated Testing. How would you handle a project where your team is not that competent but you still need to deliver it on time?

4) Final round (face to face) –

a) Architectural Discovery – When you analyze an Architecture, what is the checklist you check? How do you ensure Performance and Scalability? How do you ensure Maintainability and Extensibility? How do you ensure Reliability?

b) Architectural AlternativesРExplain the architecture of your current project (here again Design flaws, if any, will be discussed Рreason for choosing an asynchronous architecture over synchronous etc) and suggest some alternative architecture(s), stating the pros and cons of your alternative architecture(s).

c) PoCs – When and why do you need to perform PoCs?

d) Patterns – Have you developed something in any of your earlier Project which you can say is a Pattern (Architectural or Design) of it’s own? Have you refactored any existing pattern to suit your project needs?

e) MS Technology you don’t like -Which MS Technology you don’t like and why? Given a¬†chance to improve that technology, what suggestions you would give to MS to improve it? How would you actually architect/design/implement those suggestions?

f) MS Design/Code Refactorings – Have you ever felt that if MS could provide “this offering” in the Core .NET Framework or any other Frameworks, it would be great? What are those offerings and why do you think those offerings would be beneficial? What suggestions would you give MS about how to design/implement those offerings?

g) Risks – What type of Risks (Technical, Process or Delivery) did you face in your Project(s) and how did you mitigate those risks?

h) Value Added Service – Has there been a scenario wherein you proactively suggested the client to follow approach A (say some Architectural or Design Decision) rather than following approach B? Basically, how did you add value to any of your Project by going against the wave? How was your suggested approach better (architecturally or design wise) than the other approaches?

i) Strength and Weakness – What are your strengths and what are your weaknesses? How do you handle your weaknesses and your over-strengths?

j) Why MS? – Why do you want to join MS? What do you think, why MS should hire you?

k) An Idea can change yours or someone else’s life РGiven 1M$ to you, how would you invest it in some unique idea which will use technology? Details of the idea and how technology will be beneficial to implement that idea. How would that idea be beneficial for the Society as a whole? Need to tell about the technologies that will be used and rationale behind the Architectural/Design/Implementation choices to implement the idea.

That’s all I have for now.

P.S. -> I think the MS interview process has been one of the best experiences in my career. They not only test your technical skills(depths as well as breadths, to certain extent) but also your creative and imaginative skills. And I think most of the Software/IT firms should drill candidates in some similar way and why only firms, in fact, it’s beneficial for the candidates as well since such a process ultimately leads to their(including mine) overall growth.

And by the way, if you are a Computer Science fresher and want to join firms like Microsoft or Google or the like, you need to be very well prepared in Mathematics for Computer Science(questions here might be asked in the form of puzzles), Data Structures, Algorithms and basics of Distributed Computing for sure.In fact, these streams lay the foundation for AI(Artificial Intelligence) and it’s allied branches. Some firms test your skills in other areas like SDLC, Estimation(e.g. How many windows are there in India? – don’t worry, they just want to check your approach/thought process, you need not know even the approx number since they also don’t know that ūüôā ), C Programming/C++ Programming, DB Design, SQL and¬† Software Design (OOAD/Design Patterns/UML) (as far as this link is concerned, you can go through the resources mentioned in the Basics and Design Patterns sections to have some level of understanding of OOAD and Design Patterns, skipping the Inversion of Control/Dependency Injection part, since as a fresher, I don’t think you need to know that but if you have interest you can¬† – for UML you can refer UML Guide from Source Making while for questions on UML, you can refer UML Interview Questions Part 1 and Part 2). And by any chance, if you have interest in Computer Science in general and want to have basic understanding of almost all the breadths, you can refer NANDToTetris site or Invitation to Computer Science (I love the title given to this text – another text I love, atleast by the name given to it is How would you move Mount Fuji – God knows from where on Earth they come up with such an intriguing title for a puzzle book and by the way, this is also a form of creativity and imagination ūüôā ).

A Technology (Microsoft Technologies) Architect’s Confession

Whatever I have explored and experienced, a Technology/Solution Architect (Microsoft Technologies) or a good Developer as per Industry Standards, needs to be proficient in as many areas as possible as given below(I think I am still lagging in lots of the areas and that’s the confession, but I am trying to cope up as well) –

1) Basics – Being strong in Basics like Lateral Thinking, Mathematics For Computer Science, Data Structures, Algorithms, Core Framework(C#/VB.NET/F#/Java/Ruby/Python/Perl/Scala/Haskel/Lisp/Go/Prolog/SmallTalk or whetever XYZ language you can think of), OOAD, Design Patterns(including IoC/DI and Concurrency Patterns), UML, RDBMS(Data Modelling, along-with DW Design and writing really well performing complex queries), ORMs(Entity Framework,Hibernate etc), Meta/Functional/Dynamic/Reactive/Multi-Threaded/Parallel/Asynchronous/ Distributed/Logic Programming concepts, MicroServices(Microservices Architecture – another alternative is Self Contained Systems) and SOA(both SOAP & REST – an intriguing read on Micro-services is Reactive Cloud Actors: no-nonsense MicroServices), Enterprise Application Architecture(the Repository Framework is based on concepts learned from Martin Fowler’s Patterns Of Enterprise Application Architecture and .NET Domain Driven Design with C#), Enterprise Application Integration (using approaches like ESBs, MQs, Flat Files, Databases etc – an excellent paper on dealing with Transactions in an Integration environment is Your Coffee Shop Doesn‚Äôt Use Two-Phase Commit), Domain Driven Design alongwith Naked Objects(including CQRS and Event Sourcing – an interesting read on DDD is Patterns, Principles, and Practices of Domain-Driven Design while some intriguing reads on CQRS are¬†Edument CQRS and Intentful Testing Starter Kit, Exposing CQRS Through a Restful API, Building a Reactive DDD and CQRS Based Application Using Akka and More on CQRS and Actors), BPM/Workflow(BizTalk, WWF, jBPM etc, including Workflow Patterns – some good reads on workflow are Pattern based analysis of Windows Workflow, Pattern Based Analysis of WS-BPEL and WWF, Uderstanding K2 and Windows Workflow), Rules Engine(Drools/Drools.NET, MS BRE, WWF Policy Activity, WWF Externalized Policy Activity and Nuget Rules Engine Packages– infact, Rules Engine based approaches can be used to build Event Driven Systems or Calculation Engines), Schedulers/Watchers(Windows Service,Quartz etc – If you have dependency on Windows features, like logging, eventing, access to windows resources- go windows scheduler/windows services route else if it is just db to db movement or if you need some kind of heavy db function usage- go SSIS route),¬†Synchronization Frameworks(like MS Sync Framework), CEP(Complex Events Processing), Web Technologies(SPA,MVC etc), Client Side Technologies(HTML, CSS, Object Oriented Javascript, JS Frameworks like JQuery, React.js, Mustouche.js, Angular.js, Knockout.js, Require.js, Underscore.js, Sammy.js etc – one nice article, explaining the associated high level patterns can be found here while Designing Interfaces is a text one can go through to have a gist of some common UI Patterns), Real Time Web(using server side push technologies like ASP.NET SignalR,Comet etc –¬† a nice explanation of a ASP.NET SignalR based implementation can be found here but probably the best resource to understand all the intricacies of ASP.NET SigNalR is SignalR Programming in Microsoft ASP.NET), Rich UI(Windows Store & WPF, HTML5 etc), Enterprise Mobility(Android,iOS & Windows Phone), Node.js(MEAN Stack, Node.js Design Patterns, Isomorphic Javascript, To Node.Js or Not To Node.Js,¬†Getting Started with Node.js for the .NET Developer, ¬†Edje.js for Node and .NET together, Converting a Web Application from ASP.NET MVC to Node.js, ¬†Google’s V8 JS Engine in .NET etc), Enterprise Portal and ECM(Top 5 .NET Based CMS, Sharepoint, Documentum,Filenet etc – some overall good articles on Sharepoint development in general seems to be When To Use SharePoint and When, ASP.NET vs SharePoint – which one is better for web developers?, Sharepoint Development Building Blocks, Designing Your Taxonomic Hierarchy), CRM(SalesForce, MS Dynamics, SugarCRM¬† etc – another space to watch for is ERM), Software Security(including Claims based Identity and Access Control), TDD & Unit Testing (MSTest, Moq, FluentAssertions etc – probably the best resource to learn TDD and Unit Testing is The Art of Unit Testing while a good paper on EAI Testing is Test Driven EAI and another nice article is Unit Testing Async Code),BDD(BDD with Specflow and ASP.NET MVC & Specflow and CodedUI), XML Technologies, Aspect Oriented Programming etc.

Some other “Nice To Haves” are knowledge of BPMN, Archimate, DSLs(very essential for implementing Symbolic Programming constructs – some examples of DSL are JACE .NET and Regular Expression Parser – probably the best resource to learn building any kind of DSL is Language Implementation Patterns, written by the ANTLR author,Terence Parr), Cloud Computing(Windows Azure,AWS,Google Cloud etc), NoSQL DBs (Data Modelling Techniques shows the overall idea of how to think and design the NoSQL way while Polyglot Persistence, Big Data and Designing Data Intensive Applications seems to be some other good texts explaining the whole gamut of the intricacies of WHYs and HOWs of the current state of the art Distributed Data Technologies – some nice articles on NoSQL Movement are What is Hadoop? What is MapReduce? What is NoSQL?,¬†An Introduction to NoSQL Patterns, NoSQL DB Patterns, NoSQL DB Patterns and Polyglot Persistence and¬†NoSQL and Big Data Design Patterns), Big Data & Analytics, IoT(Internet Of Things), Artificial Intelligence(including Knowledge Representation, Web & Enterprise Search,Machine Learning,Natural Language Processing, Computer Grahpics & Computer Vision,Computational Advertising,Fuzzy Logic,Deep Neural Networks,Evolutionary Computing etc – a very vast topic in itself but makes possible some of the coolest applications on Earth), GIS(Geographical Information Systems – some good resources for the same can be found here) etc.

N.B. –
  i) Awesome.NET Рto have a collection of awesome .NET libraries, tools, frameworks and software, visit this Git-Hub repository.
  ii) A very basic Thumb Rule to follow while designing any software in general-
Keep your static parts(rarely changing) separated from your dynamic parts(modifiable or extensible parts) Рthis separation could be at the code or component or system level.If you meticulously analyze any S.O.L.I.D principle or some Design Pattern(including IoC/DI or some Concurrency Pattern) or some Architectural Pattern, the basic underlying design principle followed is to separate out the static parts and the dynamic parts.This ultimately leads to better maintainability,extensibility, re-useability, separation of concern,reliability(being better testable) etc.In-fact, this principle can be extended to design Data Models  or to implement maintainable stored procedures as well. By the way, at the application code level, the static parts, if needed, can be separated out into different classes(may be using partial classes Рalthough the partial class approach is very .NET specific) for better modularity.
¬†¬† iii) Big Data, Analytics and IoT(Internet Of Things) –
The above topics are quite booming nowadays and probably will rule the IT world for quite sometime.
But, to really understand Big Data at it’s core, the underlying foundations of Distributed Computing should be very clear (including knowledge of¬† P2P network, associated concept of Distributed Hash Table, and BlockChain which enables¬†building systems that require trust over decentralized networks using cryptographic techniques – a good .NET P2P implementation can be found here and good .NET DHT implementations are like NChord and Rhino DHT while a list of applications of P2P Networks can be found here, here and here and again some best of the best resources related to BlockChain can be found here and here). Some awesome resources to have a rock-solid understanding of Distributed Computing can be found here(alongwith the Operating System resources suggested in the aforementioned link, some other good resources to understand Operating System can be found here). These fabulous¬†Architectures (alongwith these Medium Scale Publications) can be some of the best food for thought as far as Distributed Computing in the real wild world is concerned.Also, IIT Bombay’s Distributed Computing courses viz Distributed Systems, Distributed Event Based Systems¬†and Advanced Distributed Systems – Engineering a Cloud seems quite promising.
As far as Analytics is concerned, solid understanding of Data Science concepts is mandatory.Stanford’s MS Data Science Track seems promising to master Analytics. Also, go for the ChaLearn Challenges or¬†Kaggle Competitions to have a reality check of your Data Science knowledge and how to apply it in the wild.Cloud Based Learning Services (i.e. AI as a Software Service) are worth researching as well. Also read this excellent Interview with data scientist and top Kaggler, Mr. Steve Donoho(I loved the answer to the question – “I have been working in Analytics Industry for some time now, but am new to Kaggle. What would be your tips for someone like me to excel on this platform?” – it’s not competition & winning but it’s just having fun and learning ūüôā ).
Now let’s take our footsteps towards IoT – some cool resources to understand IoT Architectures are WSO2 IoT Reference Architecture, IoT-A Site, Microsoft Product Mapping for the Internet of Things. Microsoft IoT site, Microsoft IoT Blog and A Map of The Internet of Things Market are also worth going through.
¬† iv) AI(Artificial Intelligence) basically consists of –
a) Knowledge Representation – formulated using some advanced Data Structures.Semantic Web is one such advanced Data Structure used to represent the knowledge associated with Web.Some good reads on Semantic Web and Ontology are Simple Ontology Support for C#, Music and the Semantic Web and Fuzzy Ontology Framework.Some good open source projects related to Semantic Web are dotNetRDF, WordNet.Net and Fise.
b) Search & Planning – implemented using some advanced Back-Tracking or Branch and Bound algorithms.
d) Reasoning(through Data) – based on concepts of Mathematical Logic(Predicate Logic,Propositional Logic etc) and Fuzzy Logic.
-> A Side Note –
AI and it’s related branches seems to be best explained by the AI Guru John Platt (ex Distinguished Scientist at Microsoft Research, currently working as Principal Scientist at Google) in the posts viz. Microsoft’s John Platt Talks About Artificial Intelligence and Machine Learning Trends from NIPS 2014. To know more about current Deep Learning Research trends, follow the Deep Learning Guru Yann LeCun (Director of AI Research, Facebook).
But, before digging into the depths of AI, I would suggest to have a solid foundation of Mathematics for Computer Science including¬† Algebra, Number Theory, Geometry, Combinatorics, Graph Theory, Mathematical Logic, Linear Algebra, Multivariable Calculus, Mathematical Programming/Operations Research/Optimization Theory, Probability & Statistics, Game Theory(Game Theory Resources – I and II), Auction Theory etc, Data Structures and Algorithms since most of the practical aspects of real world AI apps needs solid underlying Mathematics which is implemented using some basic/advanced Data Structures and basic/advanced Algorithmic Techniques.Just Enough Math¬†can be helpful here to some extent. Other resources like Data Science Starter Kit and Machine Learning Starter Kit can take you to the long mile.Basically, you need to be a very good Mathematician as well as a very good Algorist which I am still nowhere near to and still struggling ūüė¶ . And ofcourse you need to have good understanding of Distributed Computing if there is a need to implement customized elastic scalability of AI apps(else Cloud is there to save you). Probably, after that, you can go through Carnegie Mellon University courses to have a gist of¬†capabilities of AI and it’s related branches.A very good resource dealing with almost any thinkable aspect of AI is AI Topics.Industrial and Engineering Applications of Artificial Intelligence and Expert Systems seems to be some other promising reads.
¬† v) CEP(Complex Events Processing) & Big Data –
Although there are n number of tools/frameworks(e.g. Esper in Java world and StreamInsight from Microsoft) to support CEP, but probably the best option to solve CEP problems is Disruptor which uses Ring Buffer concept to process millions of records in seconds. Martin Fowler has very well explained about Disruptor here.Another good explanation of Disruptor can be found here.Storm seems to be another promising framework which seems to have more market share(being able to ease Distributed Events Processing) compared to the Disruptor but Storm also internally use Disruptor as mentioned here.SharpStorm seems to be the closest(partial) .NET implementation of Storm(another nice introduction) although one can opt to use Apache Storm on HDInsight as well.A very nice article describing Complex Event Processing and associated concepts is Stream Processing, Event sourcing,Reactive,CEP….and Making Sense of it All.Some good articles on Stream Processing in general can be found at the Confluent Blog (Confluent is a start-up founded by some ex-employees of LinkedIn). An architectural style related to Stream Processing(& Big Data) in general is Lambda Architecture and one of the best articles I could gather from the net regarding this architectural style is Lambda Architecture: Design Simpler, Resilient, Maintainable and Scalable Big Data Solutions. Emerging Trends in Big Data Technologies which is worth reading as well.
  vi) AOP(Aspect Oriented Programming)-
Gurus might suggest that AOP ultimately lowers the performance of the application since most of the AOP frameworks use code interception/injection using some sort of reflection or some similar technique.But one really useful usage scenario for higher regions(SIT,UAT,PROD etc) can be Tracing of method parameter values and method returned values. Sometimes we find that we get some hard to debug issues in higher regions and think that if we could have the traces of all the values that went into some method(s) and values returned from some method(s) very quickly – atleast for SOMETIME – then the issue could have been resolved much quicker.Well this is basically Tracing and we can keep log of such traces by writing some normal tracing module and calling the tracing module api(which will log the traces in some file system or DB or whatever) in each and every method.Yuck! That’s very hectic.Well, don’t worry, be happy.Because there is AOP to the rescue.Just write some nominal code proactively while developing the app code and incorporate AOP in the app and that’s all. Well is it really that’s all??? Well not yet.As mentioned earlier, having AOP in place for the whole life cycle of¬† the running application can degrade the performance.Well, again, don’t worry.There are lots of AOP frameworks in the market which lets you configure to use or not to use AOP(while the app is running) just by setting some config values.So when you need to enable Tracing to debug¬† some issue in higher region, go for AOP else just keep it disabled wherein no code interception/injection will be there and performance will be normal.

By the way, that’s not the only usage of AOP. In general, AOP (sync/async) techniques can be used for any instrumentation stuffs like Web Analytics, Health Monitoring etc. In fact, AOP techniques can be used for implementing Caching, Validation, Authorization, Performance Profiling and Exception Logging as well.

¬† vi) Enterprise Architect, The Awesome tool – Probably the best tool to use to construct any diagram,be it UML or BPMN or some other Architectural diagram, seems to be Sparx System’s Enterprise Architect.Although there is Visio or Visual Studio Ultimate Edition’s Architectural features(very .NET specific), but when you compare these tools with Enterprise Architect, it seems as if you are comparing an ant with an Elephant.
2) Review – Get familiar with Code Review Process(as per Coding Standards in general and specific to Project). If needed download some good Coding Standards document from internet and use that.Get familiar with Static Code Analyzers like Visual Studio InBuilt Tools,NDepend,SONAR,FxCop etc.If possible use Resharper and if needed go for a Reflector as well. Visual Studio 2013 Static Code Analysis in depth gives you an in-depth analysis of most of the Static Code Analysis techniques available surrounding Visual Studio environment.
    Sometimes you may need to review code from a Functional point of view as well(so, understand well, the domain as well) or may need to do application specific code comparison as well(especially, applicable for Re-Engineering Projects). While at other times you may be asked to do optimization based review i.e whether the code/design is optimized with respect to re-usability, extensibility, maintainability, testability, security, performance etc.
    Basically, while doing any review look for project best practices like Coding Standard, Architecture, Governance, Automated Build & Deployment, Change/Release Management(Request For Change, Release Notes, Run Book etc),  Unit testing, TDD, Cyclometric complexity(CC) report, Check in policies, Environment readiness, Design for scale, if they have done/doing POC to validate or not  etc..
 N.B.-> Some good resources for Static Code Analysis(a Language Engineering aspect of Computer Science) in general are Static Code Analysis, Secure Programming with Static Analysis, Techniques for Program Analysis and Verification and Semantic Designs. Tools For Software Engineers (Microsoft Research) also seems promising.
¬†3) Performance – Get hold of Profiling tools(NP .NET Profiler, Visual Studio Profiler,RedGate’s Ants Profiler etc) to profile application/database memory usage & performance.Learn .NET Application and Database Performance Tuning. For OLTP based systems, TPM-C benchmarks are quite good.
N.B. – Certain high level solutions related to Performance issues are –
¬†¬†¬† a) Caching – If it’s a Web Farm/Garden environment, you should opt for a Sticky Session approach or some Distributed Cache approach. A very .NET specific tip is to avoid using HTTPRuntime.Cache or HTTPContext.Current.Cache for a clustered deployment using Distributed Cache approach but for a Sticky Session based approach one can use HTTPRuntime.Cache(over HTTPContext.Current.Cache). Some cool implementation approaches for Caching can be found here and here.
    b) Reactive,Parallel and Asynchronous Computing(not necessary that these paradigms needs to applied w.r.t only multi-core machines but rather can be applied in a totally distributed environment using commodity machines as well РHigh Performance Computing takes the distributed approach using multi-core machines).
    c) Hardware Scaling Рboth Vertical(increasing the RAM size etc) and Horizontal(increasing the number of machines and using a load balancer Рthis approach can be used for Disaster Recovery as well using Fail-over Clustering)
    d) Database Performance Tuning(probably one of the biggest topics with issues related to performance Рan ocean of knowledge in itself).
    e) Another option (as far as OLAP systems are concerned ), can be, to use an amalgamation of NoSQL DBs(enables super fast data retrieval) and Hadoop(a very high performance computation framework) Ecosystem.
    f) And ofcourse, any combination of the above mentioned performance tips.
    g) A very .NET specific performance tip is to use Interop Services and Reflection code as less as possible.As far as Reflection is concerned, try to use some caching mechanism alongwith using DynamicMethod or Expression Tree wherever possible.
¬†N.B. -> Some awesome resources w.r.t Performance tuning/testing are –
       (very .NETish)
    c) Performance Testing Guidance
 P.S.-> Some good pointers to Client Side Performance Profiling Tools are Client side performance and Profiling JavaScript  With Ajax View Tool. For Web Apps/Services, one should be conversant with Fiddler.To know how to use Fiddler for Performance Testing, one can refer Building Faster Sites and Services with Fiddler.
4)Documentation – Be strong in documentation(HLD and LLD).Get some good HLD(NFRs ,Cross Cutting Concerns,Physical/Deployment View,Logical View,Dynamic view having sequence and activity diagrams,Backout Strategies etc) and LLD samples of large/complex Web apps and large/complex BPM & EAI based Apps from the internet and understand the WHYs and HOWs.Have one HLD and multiple LLDs as per the number of modules in the Application.
 5) Estimation РBe strong in Estimation Рprobably the most trickiest part in SDLC. Be conversant with Estimation techniques viz. Work Breakdown Structure, Function Point and Use Case Point. Consider the whole of SDLC while estimating mission-critical Applications.
Certain things to keep in mind while doing any estimate –
Break the work tasks into as much detail as possible and then give the effort estimate(it’s always easier to give the estimate by following this simple approach).Certain things can be re-used, take those also in consideration.Break up the estimate into different phases of SDLC – e.g. User Stories or Screen Shots or Code Analysis(reverse engineering the code to extract the requirements can also be a scenario – sometimes if you just give the estimate based on going by the screens, you might fall into a big trap – very meticulous and thorough analysis is the key) or all of these,Designing as per new Technologies,Construction and Manual and Coded Unit Testing.Scale(may be out of 5) the complexity of the requirements (if possible try to give the rationale behind the same) and also have a scale(again may be out of 5) for resource competency and productivity and based on that do the estimate.Provide all these inputs in some excel sheet to make it more understandable.If any of these are missing, then please ask the team to prepare the estimate on these detailed lines and prepare the excel sheet accordingly else it’s very difficult for an individual to review an estimate independently.All these guidelines basically makes it easier for a reviewer to know the basis(rationale) behind the estimate and also makes the estimate more justifiable to the client.
Be very much aware about the scope.Some other factors like Code Analysis,Code Refactoring,Code Merging and the whole gamut of Application Lifecycle Management’s associated approval process should also be taken into consideration while doing any estimate.Separating out the must haves and nice to haves can also reduce your estimate by quite an ample amount of effort.
For Agile based estimation, one can refer Martin Fowler’s Estimation articles.
N.B.-> Some good articles to point to the factors based on which one should opt for Function Point or Use Case Point Estimation are РDear Dr. Use Case: What About Function Points and Use Cases?, Function Points or Use Case Points?, Function Points, Use Case Points, Story Points
N.B.-> A Random Thought on Estimation ->
Accurate estimation of software development effort is critical in Software Engineering. Underestimates lead to time pressures that may compromise full functional development and thorough testing of software. In contrast, overestimates can result in noncompetitive contract bids and/or over allocation of development resources and personnel.So basically, any Project Estimation is a problem related to Optimization Theory in general and Machine Learning is a great tool to solve most of the real world Optimization problems and Project Estimation is no exception.Agreed that too many input parameters are there for a Project Estimate(alongwith changing requirements) and you need a humongous amount of data to practically predict some estimate for a project using Machine Learning tools but it’s not impossible, as suggested here(the techniques may not be applicable for all the scenarios – it all depends on the context of estimation). Some good pointers to apply Machine Learning to Software Engineering in general (including Project Estimation), seems to be Machine Learning Applications in Software Engineering, Advances in Machine Learning Applications in Software Engineering, Software Engineering with Computational Intelligence, Computational Intelligence in Software Engineering etc.
¬†6)Frameworks – Get acquainted with new UI and Application Frameworks and adapt to them ASAP.If needed, do some PoCs.Also if needed, try to develop Frameworks for custom needs(on your own). Also, all of a sudden, you might be asked to extend custom 3rd party Frameworks of which you might not have any knowledge.The problem with such Frameworks is the Learning Curve and it’s also probable that you won’t get any help from google for such Frameworks.Only the documentation(if any) and probably GOD can save you :). Please assess the associated Risks,Assumptions and Issues here. Some examples of Business Application Frameworks are like CSLA.NET and Griffin.Framework.
 7) BD(Business Development) ActivitiesРMight need overall solutioning knowledge rather than just knowledge of .NET(or other language) specific technologies to develop the response for an RFP(Request For Proposal). Also, should be acquainted in preparing presentations for RFPs and making the stakeholders understand the same.This might need to get involved in Pre-Sales activities alongwith the Business Analysts.Also, may need to understand the domain in totality and develop the Business Architecture alongwith developing the Information Architecture(Data Modelling,Cleaning,Analytics etc). Also there should be considerations for the associated Risk & Issue Assessment and Management. Get involved in Pre-Sales activities.Some good resources to write proposals for responses to RFPs seems to be Handbook for Writing Business Proposals and Persuasive Business Proposals. Also, the Communication Skills Course from IIT Bombay can be helpful here.
¬†8) Architectural Discovery and Assessment – That’s another area that needs to be concentrated.Need to have a 360 degree view of existing Architecture followed by remediation(if any) to minimize the Technical debts.A good start related to all these can be found here. Knowledge of Assessment related Planning and Governance is also a good to have.
 9) Development Architecture(Application LifeCycle Management) РUnderstanding the whole gamut of ALM(Application LifeCycle Management) is a must.Some good resources for ALM using Visual Studio and TFS can be found here. Benefits of using an ALM software can be found here.Other good resources are like VSARGuidance and VSARBranchingGuidance.Also the almspecialisttoolkit mentioned  in the Microsoft Open Source Project Directory is cool. Microsoft ALM Build Futures also seems quite promising.Git Support for Visual Studio is a good read as well.The knowledge of Agile Process is another mandatory stuff in the Development Architecture area.DevOps is another area to look out for.Some promising resources on DevOps seems to be The Phoenix Project and Continuous Delivery.
 10) Infrastructure Architecture  РUsing some thrid party Capacity Planning Framework or developing some in-house Capacity Planning Framework(alongwith Hardware Sizing Рmainly to decide the disc space and RAM size required) for different Virtualization options  can lead you to an edge.Basic NFRs(Non Functional Requirements) to decide on Capacity Planning(alongwith Hardware Sizing) for a Web Application can be something  like Number Of Concurrent Users who will access the application and the SLA(Service Level Agreement) for the processing time to process a page request.

11) Software ArchitectureMost of everything mentioned above mainly applies to a Application Architect/Developer but in general, as an Architect (more specifically, a Systems Architect), you are supposed to define the System Architecture as well, where it’s a prime importance that you pursue the qualities to plan and execute NFRs (Non Functional Requirements). Some good resources in this space are Software Quality Attributes, Functional or non-functional requirement?,¬† Understanding Non Functional Requirements, Non-Functional Requirements (NFR) in Agile Practices, Non Functional Requirements – Minimal Checklist,¬†Non Functional Requirements in Software Engineering, Four Layered Approach to Non Functional Requirements Analysis , NFRs – Practices and Recommendations, Quantifying Non-Functional Requirements: A Process Oriented Approach and Architecting High Performing, Scalable and Available Enterprise Web Applications.In-fact, OCMJEA¬† is indicative about the same as indicated here(Phase 3 section). A .NET Developer/Architect can map the OCMJEA specific technologies to .NET(a simple google search can help here) and learn some cool stuffs accordingly.Also as a .NET Systems Architect, expectation w.r.t. planning and execution of NFRs can be for a non-Windows environment as well since lots of viable options have come up as indicated here and here.

Some good resources w.r.t. Software Architecture(& more specifically, Systems Architecture) seems to be Just Enough Software Architecture, Software Architecture for Developers, Software Architecture in Practice, Software Systems Architecture, Documenting Software Architecture, Evaluating Software Architectures (including Tools and Methods for Evaluating the Architecture), DevOps: A Software Architect’s Perspective, Architecting the Cloud, OReilly Software Architecture Series, 97 Things Every Software Architect Should Know, ThoughtWorks Anthology(Vol 2) and Beyond Software Architecture.

An overall nice read is Enterprise Architect vs Solution Architect

A few examples of users getting their accounts hacked: Paypal & GoDaddy, Apple & Amazon, Twitter Employees. Use two-factor authentication for important accounts, especially email accounts because other services’ password reset procedures use email addresses.
A side note on Architecture ->

Architecture involves studying the requirements, choosing architectural style, patterns e.g. layer, MVC, broker, Microkernel. Essentially its about the organization of subsystems in a system and interaction among themselves. Apart from business requirements, quality attributes like performance,availability, scalability, reusabilty, modfiability, testability etc need to be considered as they influence the architecture directly. The inputs for architecture are business requirement, use cases/ stories while output are contextual view, implementation view, process view along-with bench-marking for quality attributes and ATAM (Architecture Tradeoff Analysis Method). Are you a Software Architect can throw some more light in this space.

12) Innovative Thinking – Although this quality applies more to a Scientist/Inventor, but such thinking can be a boon not only for you but for the society as a whole e.g. Pranav Mistry’s Sixth Sense technology is one of the most innovative thinking which is going to shape the future for the betterment of the mass society as a whole.These are some capabilities of Human Computer Interaction and other Possibilities can be found here. The¬† initiatives from Apache Projects,Microsoft Research, Google Research, Facebook Research/Facebook Engineering, Netflix Technology, LinkedIn Engineering, Twitter Research/Engineering, Yahoo Research/Yahoo Engineering, IBM Research etc.¬†can be helpful to get ideas on the areas¬†where Innovative Thinking is currently applied.Also worth is to keep a focused watch on Gartner Hype Cycle (although here it’s pointed to the 2015 edition but try to keep a close watch on the latest edition). EIT Digital and MIT Centre for Digital Business also seems quite promising in research in the Digital Space.
 13) Soft Skills
a) Ask questions Рvery important for a consulting job like that of an Architect РAsk Functional as well as System level questions.You might have very good technical in-depth knowledge but if you solve the problems only after you come to know about them, then that is not of much use.Use your technical in-depth knowledge to ask proactive questions relevant to the context.Try to analyze and think about the existing pain points and problems and ask questions accordingly.
b) Enabling a Team ASAP and applying “Divide and Conquer” Principle wherever possible.
c) Be Proactive in everything rather than being Reactive.
d) Try to have a structured(sequential) thought process to solve any kind of problem.
e) It’s important that you have as much knowledge of Technology(Java, .NET, AI, Data Science, Big Data, Software Architecture etc) or the Digital Transformation Movement but probably it’s most important to think about your Client’s Clients i.e. Customers. Always try to make the Customer Experience as better as you can, with whatever innovative ideas you have(applying Digital Technology maybe).I lag this arena by miles though ūüė¶ . Some case studies showing how firms enhanced their Customers’ Experience via Digital Transformation can be found here.
f) And most important – “Think before you act”.
N.B. -> Perhaps, with respect to the soft skills mentioned above, Human Dimensions of Organisations can help you here.
P.S. ->
b) Some very good resources w.r.t. solution approaches to solve different scenarios using Microsoft Platform are Applied Architecture Patterns on the Microsoft Platform (2nd Edition) and Microsoft Application Architecture Guidance. To have a gist of the capabilities offered by the Microsoft Stack one can go through What to Use on the Microsoft Stack.Also worth looking at are Microsoft .NET – Architecting Applications for the Enterprise, 2nd Edition, Architecting Mobile Solutions for the Enterprise and Microsoft Servers and Enterprise Development.White Papers by David Chappell are also worth going through.
c) Some very good resources (from Microsoft) on specific Architectural paradigms are Enterprise Solution Patterns using Microsoft .NET, Microsoft Integration Patterns , Microsoft Data Patterns and Windows Azure Guidance.
e) Also, one may opt to go via the route of TOGAF(How to prepare for TOGAF) or the Zachman Framework or FEAF to formulate an Enterprise Architecure. A good overall resource in this arena seems to be Enterprise Architecture Patterns. Enterprise Architecture Certification also seems quite promising.
f) ThoughtWorks’ Technology Radar and Insights are quite helpful in deciding the technical options to choose for a certain use case.Keep a watch on O’Reilly Radar as well.
g) Whatever mentioned in this article are views based on going through some really good resources and¬† interactions with some beautiful minds at my workplace.If someone has some other view(s)/suggestion(s) then please leave a comment :). And please pardon me for some formatting issues above. There seems to be some problem with the WordPress Editor which I am not able to resolve ūüė¶ .

Kool Resources for Domain Specific Languages

1) Domain Specific Language in .NET

N.B. РJACE.NET, State Of the Art Expression Evaluation, Building Expression Evaluator using Expression Trees in C#, C# Expression Evaluator, ANTLR C# Grammar, Grammars and Parsing in C# 2.0, Writing your first Domain Specific Language, Irony Source Code, Creating your own Domain Specific Language, Regular Expression Parser in C#, Gold Parser and Spruce: From Irony.NET to GoldParser.

2) Language Implementation Patterns: Create Your Own Domain-Specific and General Programming Languages

N.B. -> The Definitive ANTLR Reference: Building Domain-Specific Languages

3) 4 practical uses for Domain Specific Languages

N.B. -> Building Domain Specific Languages in C# – Part 1 and Part 2

4) Creating Domain Specific Languages in C#

N.B. -> Developing a Complex External DSL , Building an External DSL and Building an External DSL in C#

5) Domain Specific Languages

N.B.-  DSL Patterns & Language Workbenches: The Killer-App for Domain Specific Languages?

6) DSLs in Action

7) DSLs in Boo РDomain Specific Languages in .NET

8) Domain Specific Languages РVisualization and Modeling SDK

N.B. -> Domain Specific Language Tools , Software Factories , Practical Software Factories in .NET

9) Awesome Compiler Design and Construction Resources(or some related topic) –

i) Introduction to the Theory of Computation

ii) Programming Languages : Principles and Practice and Compiler Construction: Principles and Practice

iii)  Compilers: Principles, Techniques, and Tools (Dragon Book)

iv)  Modern Compiler Implementations in Java (Tiger Book)

v)  Advanced Compiler Design and Implementation (Whale Book)

P.S. -> Codeplex DSL Projects¬†and .NET Compiler Platform (“Roslyn”)

Resources for Microsoft Technologies

1)¬†Basic¬†Resources –

a) Code Complete and Fundamentals of Computer Programming with C#

N.B.-> Learn C# by Building a Simple RPG, O’Reilly’s Coding4Fun Text , Channel 9’s Coding4Fun (Toolkit) and Java Projects that will Have You Inspired and Sharpen Your Skills(think and try all these out in .NET and I bet, you will learn a hell lot)

b) C# 6.0 in a Nutshell

N.B. -> Briefly Exploring C# 6.0 New Features

c) C# 4.0 How to

N.B. -> Ben Watson Talks about C#, WCF, Manycore, and Big O

d) C# for Financial Markets

e) C# in Depth

f) .NET 4.0 Generics

g) .NET and COM: The Complete Interoperability Guide

N.B.-> Create a Custom Marshaling Implementation Using .NET Remoting and COM Interop

h) CLR via C#

N.B. -> Customizing the Microsoft .NET Framework Common Language Runtime

i) Metaprogramming in .NET

N.B.-> Probably, the best part about the above text is Chapter 6.Why? Well, it gives insights into building Genetic Algorithms using .NET Expression Trees.

j) Mastering Regular Expressions

k) Programming .NET Components : Design and Build .NET Applications Using Component-Oriented Programming

l) C# Network Programming and Network Programming in the .NET Framework

N.B -> Dynamic vs. Var vs. Object, Creating a .NET Windows Service: Three Different Approaches, Managed Threading Best Practices,¬† Using Expression Trees in Your APIs, Expression API vs Reflection, Buffer,File and Stream, CLR Inside Out Series , Coding Bibliography ,¬†Value vs Reference Types in C#, List Passed By Ref,¬†2,000 Things You Should Know About C# , .NET Extensions Methods Library¬†, ExtensionMethod.NET , AutoMapper ,¬†Clone with Better Performance, .NET Deep Copy, Improving C# Reflection Performance , Strongly Typed, High Performance Reflection with C# Delegate ,¬†Dynamic Type using Reflection.Emit , Get Drunk on the Power of Reflection.Emit , Destructors in C#(Using the Destructor and IDisposable together) and Dispose Pattern, Understanding Application Domains , Executing Code in a Separate Application Domain Using C# , Loading Native (NGEN) images and its interaction with the GAC, Exploring and Manipulating the GAC, Demystifying the .NET Global Assembly Cache, GAC Phobia, Anything wrong with NOT signing a .NET assembly?, Why sign an assembly , Strong Names , Consuming assemblies that are located in a folder different from the application base folder, How to add DLL search path, In what order are locations searched to load referenced DLLs?, Dynamic-Link Library Search Order, assembly loading from GAC or Bin (Dll in both the bin and the gac, which one gets used?), The Right Way to Version Your Assemblies, Design-Time Code Generation by using T4 Text Templates, How to have an auto incrementing version number, Using T4 templates to manage assembly version information, How do I determine the dependencies of a .NET application?(sometimes very useful to solve GAC related issues), Learning Network Programming in C#, A Complete Impersonation demo in C#, Imperatively Impersonate a User , Understanding WeakReferences in .NET , W is for …Weak Reference , Lambda , Castle’s DynamicProxy, There is no source available for the current location, IsAssignableToGenericType, Exception.Message vs Exception.ToString(), Pass a method with N params as parameter in C#, Chained null checks and the Maybe monad, .NET Remoting Static method Execution, C# For Scripting, Friendly Unique Id Generation, Can attributes be added dynamically in C#?, Change Attribute’s parameter at runtime, Choosing Between DateTime, DateTimeOffset, TimeSpan, and TimeZoneInfo and LinFu Framework – Part I, Part II, Part III, Part IV, Part V and Part VI.

2)¬†Functional Programming –

F# Books and Tutorials

N.B. -> F# Performance in Scientific Computing, Functional Programming for Everyday .NET Development, F# Programming wiki , MSDN РReal World Functional Ppogramming , Functional C# , FC# РInternal Functional DSL to C# 3.0 , .NET Functional Programming Series , FSharp.ML-industry needs(Machine Learning For .NET) and Functional Programming Bibliography

3) Dynamic Programming –

a) Pro DLR in .NET 4.0

b) DLR Project (The documentation provided, is a really good read)

c) IronPython in Action, Pro IronPython and Professional IronPython

d) IronPython Project and IronRuby Project

N.B. ->  Usage in Private Reflection, Dynamic XElement Wrapper, LINQ to CSV using DynamicObject , Dynamic Method Articles , Grokking the DLR, Scripting ASP.NET MVC Views Stored In The Database

4) Concurrent¬†Programming –

Concurrency in C# Cookbook and CLR via C# (last 5 chapters)

N.B.->  Pro Asynchronous Programming with .NET, Parallel Programming with Microsoft .NET, Introduction to Rx, Best Practices in Asynchronous Programming, Best Practices for c# async/await, Building Parallel Algorithms, Introduction to Async/Await on ASP.NET, Async in C# 5.0, Multithreading in C# 5.0 Cookbook , Threading in C# , C# 2008 and 2005 Threaded Programming, Seven Concurrency Models in Seven Weeks: When Threads Unravel, Patterns for Parallel Programming РUnderstanding and Applying Parallel Patterns with the .NET Framework 4 , Articles on Parallel Programming with the .NET Framework 4 , Parallel Programming in the .NET Framework , A Tour through the Parallel Programming Samples for .NET 4 , Specialized Task Schedulers in .NET 4 , Practical Parallel and Concurrent Programming , Professional Parallel Programming with C#: Master Parallel Extensions with .NET 4 , Pro .NET 4 Parallel Programming in C# , Parallel Processing and Concurrency in the .NET Framework , Concurrent Programming on Windows , Patterns for Parallel Programming , Introduction to Parallel Computing , Developing Parallel Programs ,  Developing Parallel Programs РA Discussion of Popular Models , Work Stealing in .NET 4.0 ,Lock Free Work Stealing Queue , Understanding the volatile modifier in C# , Volatile keyword in C#, Volatile and MemoryBarrier, TransactionScope with Parallel Extensions, Transactions across a set of Parallel Tasks, Participating in TransactionScopes and Async/Await Going deep into the abyss, Get TransactionScope to work with async / await, Asynchronous Repositories, Mocking async repository calls, The Joy of Rx: Building an Asynchronous API with Rx and IQueryable, Dataflow Book, Programming Reactive Extensions and LINQ, Reactive Extensions Projects and A WebCrawler demonstrating TPL Dataflow

5) ORM – Entity Framework Code First (Learning .NET Data Development in general) –

a) But Why the Hell do you need an ORM?

b) Getting Started with EF using MVC Series

N.B-> EF 5.x DbContext Fluent Generator for C#

c) Associations in EF Code First

d) Entity Framework 4.1: Expert’s Cookbook

e) Programming Entity Framework: Code First

f) Programming Entity Framework : DBContext

g) Writing an Entity Framework Data Provider

h) Entity Framework Metadata –

Querying Entity Framework Metadata

MetadataWorkspaceExtensions Class

Extended Entity Framework

N.B. -> Using EF Code First, for BulkInsert use a EF wrapper over SqlBulkCopy while for BulkUpdate and BulkDelete use a EF wrapper over table valued parameter. But beware of the fact that while using table valued parameter the order of columns in a datatable need to be in the exact same order as the UDDT as no column mapping is applied.Also, if you are using DBContext directly or ObjectContext(e.g converting a DBContext object to ObjectContext object) and it’s a multi-threaded application,please use it within some lock (or better use a per-thread DBContext/ObjectContext object), since DBContext/ObjectContext is not thread-safe. Whole EF is THREAD- UNSAFE.My personal preference is to use per-thread,per-BoundedContext(can be small functional modules as well) DBContext objects since this approach safe-gaurds multi-threading issues and also gives better performance.

N.B. -> An ADO.NET Tip for highly Concurrent Scenario -> Always use connection,command,data reader, data adapter etc objects using “using” construct of C#. Always open the connection only when u really hit the DB e.g calling DataAdapter.Fill,ExecuteNonQuery etc.So just before doing the actual DB hit open the connection and just after that close it – you can create connection,command etc in that order but technically, you should open any connection just before doing any actual DB hit and just after that close it.Also if there are methods like Get,Save etc in your DAL, use separate set of connection,command etc objects for each method.

Also, one can refer the Extensions Methods Library having very good EF6+ support.

i) Entity Framework Performance Considerations

j) Entity Framework Code First Samples

k) ADO.NET Entity Framework : Stored Procedure Customization

N.B. -> Stored Procedures with Multiple Active Result Sets

l) Entity Framework and Pessimistic Concurrency

N.B.-> DB Concurrency Control with .NET and 6 ways of doing locking in .NET (Pessimistic and optimistic)

m) OData Programming Cookbook for .NET Developers

N.B. -> OData by Example

n) A Tip for Writing Entity Framework Queries Using Datetime Values

N.B. -> Difference between SQLFunctions and EntityFunctions

o) Async Entity Framework Queries and Async database polling with EntityFramework 5

p) Testing EF with a mocking framework

6) Unstructured Data HandlingStructured, semi structured and unstructured data,  A Relational Approach to Incrementally Extracting and Querying Structure in Unstructured Data, Storing large files in DB, Art of SQL Server FileStream, FileStream and FileTable in SQL Server 2012, Working with Entity Framework Code First and SQL FILESTREAM, Web API File Upload with MS SQL SERVER FileTable, WCF Large Data and Streaming and Crate.IO Use Cases.

7) Enterprise Excel and VBAExcel Development using VBA, VBA Developer’s Handbook, Pro Excel 2007 VBA, Classes in VBA, Calling Sub and Function Procedures, Macros in Excel (VBA), VBA For Smarties, Excel VBA Referencing Ranges, VBA (Excel): Find Based on Multiple Search Criteria Without Looping, Cheap trick for imitating function pointer/delegate mechanism in VBA, Evaluate Functions and Formulas Fun,¬†Higher Order Functions in VBA, Application.Run, A ByRef argument with the Application.Run method, Application.Evaluate Method, Calling XLAM/XLL/Automation UDFs from VBA, Why does VBA Find loop fail when called from Evaluate?, Excel‚Äôs Application.Evaluate() does not consider all data¬†types, How to use Evaluate to invoke an Excel UDF programmatically, Complex Excel Formula in terms of Final, Excel Solver AddIn, Using Solver in Excel VBA, Error Handling via an Error Class, VBA Excel Error HAndling, VBA Error Handling, Error Handling (Global), How To Write Errors to Log File, VBA Multithreading, Multithreaded VBA, VBA Unit Testing

8) Other Good Resources in general –

a) Pro Dynamic .NET 4.0 Applications

b) Windows Powershell in Action

N.B. -> Why Powershell? and MS Script Centre

c) Manning – Continuous Integration in .NET

N.B. -> Continuous Integration in .NET – From Theory to Practice, 2nd Ed.

e) AppFabric for Windows Server

N.B. -> Microsoft Windows Server AppFabric Cookbook and  Pro Windows Server AppFabric

f) Art of Unit Testing

N.B. РIntroducing BDD, Moq, Fluent Assertions , Pex and Moles РIsolation and White box Unit Testing for .NET , Testing Strategies for WCF Services , Microsoft.Activities.UnitTesting , Fake DbConnection for Unit Testing EF Code and Testeroids Unit Testing Framework(can test asynchronous Rx and TPL)

g) Instrumentation and Monitoring РEmbracing Semantic Logging, Instrumentation with Semantic Logging Application, Semantic Logging, HangFire, File System Watcher, System Center Monitoring, AVIcode .NET Application Monitoring, Creating a System Monitoring Service in .NET , Monitor and Manage Services on Remote Machines, Monitoring MSMQ Message Queues, AppInsight Performance Analytics, Microsoft Research AppInsight, AppInsight: Mobile App Performance Monitoring in the Wild, The Top 10 Best Web Analytic Tools and Top 30 Web Analytics Tools | Google Analytics Alternative

h) Open Source Projects from Clarius Consulting Labs

N.B. -> MSDN Magazine Issues and Downloads and Microsoft Reference Source

i) XSDToCode – Awesome xsd to code generation plugin for Visual Studio

j) Configuration Transformation Tool and Microsoft Xml Document Transformation

k) NP .NET Profiler

l) Scott Hanselman’s 2014 Ultimate Developer and Power Users Tool List for Windows

Inverted Index : The basic ingredient behind the recipe called Search Engine

The wiki definition of Inverted Index is –

“An inverted index (also referred to as postings file or inverted file) is an index data structure storing a mapping from content, such as words or numbers, to its locations in a database file, or in a document or a set of documents”

Now refer the below diagram to have pictorial description of the concepts explained-

Say, I am running a website and one of the pages( is having some content .Now, a search engine when crawls the webpage corresponding to the (mentioned or any other)url and extracts the text from the webpage, it can save the url,text etc in some Database(RDBMS or NoSQL DB) or XML file etc.So the DB or the XML file (say DB1 or XML1) can be containing some structure having columns(in case of a DB) or attributes(in case of an XML file) docID,url,text ,timestamp etc.

N.B. -> Basic details of crawling and extracting text from a webpage are not provided in this post – probably I will write another post on the same – but first things first – so lets concentrate on Inverted Index.

So now what we can do is get the text from the DB or XML file(DB1¬†or XML1) and use the space separator¬†(in .NET one can use text.Split(” “)) to get all the words within the text.In addition one can get the position of the words within the text also(in .NET one can use text.IndexOf(word)). So now, we can save all these in another DB or XML file (say DB2 or XML2) having the structure having columns(in case of a DB) or attributes(in case of an XML file) word,docid,position etc.All these can be done in the background using a Windows Service (in java world something like Quartz can be used for the same – .NET port of Quartz also is available here).This background process may run on a weekly or a monthly basis.

Now when the user inputs his\her “search text” in the search box and clicks the Search button, the button click event handler can extract all the words within the “search text” along with¬†the positions within the “search text” and run a query in the DB or XML file(DB2 or XML2) to get all the docids¬†having nearest matches with respect to the words in the “search text” and their positions within the “search text”.Using these docids¬†one can run a join query in the DB or XML file (DB1¬†or XML1) to get all the respective urls¬†and the actual content.Then in the UI,all the URLs can be shown along with a trimmed version of the respective contents.

So, as can be understood, the usage of Inverted Index lets you search millions of  web documents in a lightning fast manner.

P.S. – Whatever explained here is a very basic version of Indexing(using Inverted Index concept – that happens in the background) and Querying.For a full-fledged Search Engine, other than Indexing and Querying, one needs to consider issues related to Crawling,Forward Indexer/Content Processor(such indexers/processors do¬†stemming, lemmatization, synonym expansion, entity extraction, part of speech tagging etc)¬†and Ranking(Google became famous not because of its knowledge of Information Retrieval basics – Information Retrieval knowledge was there long back then – it became famous for it’s PageRank algorithm – a solid understanding of Linear Algebra and Probability Theory is a must to understand the PageRank algorithm).Even then there are lots of other things to be considered in case of Indexing and Querying(e.g for availability ,scalability and fast retrieval of search results,one needs to deploy each of these in a distributed environment).To have a gist of how the basic architecture (and related stuffs) of a real Search engine looks like, one can go through this post by Ricky Ho.Also, here I have tried to explain Web Search in its very basic form – one can use similar concepts in Enterprise Search where one needs to do indexing(using the same concept of Inverted Index) on textual DB columns for full text search.An open source search engine¬†framework¬†that deals with most of the search stuffs is Lucene¬†(Apache Lucene.NET page). Elasticsearch is another awesome option based on Lucene. FlexSearch and SearchRoo are 2 end to end good open source samples of Search Engines developed in .NET. A very good .NET client of Solr is SolrNet.Also, check the Magic Quadrant for Enterprise Search from Gartner to have gist of the vendors in the Enterprise Search market.To have a solid understanding of Information Retrieval and associated stuffs, one can refer Stanford’s Information Retrieval Resources Recommendations.