[{"content":"As we step into the era of AI, a proper approach to problem-solving is no longer optional - it\u0026rsquo;s essential for everyone. We can choose: to tackle challenges chaotically, or approach problem-solving as a form of strategic decision-making process. In this series, I will present how to manage IT project problems by combing theory with modern AI capabilities. However, before we explore how AI can act as our \u0026ldquo;thinking partner,\u0026rdquo; we must start with a strong foundation.\nUnderstanding the Structure of a Problem Before solving any problem, it\u0026rsquo;s crucial to first understand its structure. Not every problem requires deep analysis. The level of analysis depends on several factors - the complexity of the situation, the leader\u0026rsquo;s experience, and how much time we have to solve it.\nProblems can generally be categorise three ways:\nWell-Structured - these are easiest to solve. All the information is available, and the solution is usually clear. The team knows what to do and can quickly recognise and categorise the problem. Medium Structure - the most common type. They\u0026rsquo;re partially understood, but here\u0026rsquo;s the tricky part: there can be more than one possible solution. This introduces some risk because the team might reject the right solution without even realising it. Solving these problems often requires creativity, as the team needs to explore different possibilities before landing on the best one. Ill Structure - these are the hardest to solve, because they are complex and difficult to understand. There’s often no clear solution, and the team may struggle to find the root cause. These problems involve many factors and unknowns. Identify the problem (Root Cause vs. Symptoms) After identifying the structure of the problem, the next step should be to clearly define it. I mean finding the root cause of the problem, not just addressing the symptoms. Let me give you an example - think about visiting a doctor. If the doctor only focuses on your symptoms, like pain or a cough, they might miss the underlying issue causing them.\nThe same goes for the project. If we only address surface-level issues without understanding the core problem, it’s likely to recur. That’s why it’s so important to dive deeper, using the right standards and processes to fully understand what’s happening.\nHow can we identify the root cause of a problem? To tackle problems effectively, it’s crucial to address the real root cause rather than just the symptoms. Here’s a key set of steps you can follow:\n• Define the problem’s scope clearly.\n• Ask ourselves: why did this problem happen in the first place?\n• Identify the obstacles that are keeping us from reaching our goal.\n• Write a draft problem statement that clearly describes what we need to solve and update the facts and assumptions as we gather new information.\nBy following these steps, you can make sure you\u0026rsquo;re addressing the real problem and not just its symptoms.\nGenerate Potential Solutions In this step, the goal is to create a list of possible solutions. Start by brainstorming to gather diverse ideas and perspectives. Once you have those potential solutions in front of you, it’s time to compare and analyse them.\nA helpful approach is to write a solution statement - it\u0026rsquo;s a clear explanation of how the action you’re considering will solve the problem. This statement should clearly explain how the action you\u0026rsquo;re considering will solve the problem.\nYou can also use tools like a decision matrix, Lean Prioritisation method, to organise and evaluate your options more effectively, or you can follow a more modern way that supports AI. Of course, you need to have back on your head that AI can be your thinking partner,r not the decision maker.\nBy doing this, you’ll move from a long list of possibilities to focused, actionable solutions.\nImplement the Solution You’ve Chosen Once a solutions is chosen, it\u0026rsquo;s time to put it into action. A successful implementation requires:\nClear plan - define the timeline, specific tasks, and deadlines. Assigned roles and responsibilities - make sure that each team member knows their part. Effective communication- even the best solution can fail if it\u0026rsquo;s not clearly explained. Remember: not all solutions will work perfectly the first time. Stay flexible and be ready to adapt if the initial approach doesn’t deliver the desired results.\nEvaluate Results The final step is to analyse the results, which can take weeks, months, or even years, depending on the complexity of the problem. Key actions include:\nThe team should document instructions for implementation, The team lead must monitor progress. Problem-Solving is a Continuous, Structured Process\u0026hellip; Problem-solving doesn’t end at identifying the issue - it’s only complete when the problem is fully resolved. Sometimes this means adjusting policies, refining the solution, or even restarting the process to achieve the desired results.\nEffective problem-solving is usually a structured, ongoing process, not a one-time task. You need to look at all the key factors and understand how they connect. I believe that this approach can help you uncover the best solutions and reach your goals more effectively.\nComing Next\u0026hellip; Now that we have the framework, how do we make it faster and smarter? In my next post, I will show you how to use AI tools as your \u0026ldquo;Thinking Partner\u0026rdquo; to turn messy project into clear, structured solutions.\n","date":"2026-02-16T12:28:07Z","image":"/uploads/2026/02/problem-solving-3.jpg","permalink":"/2026/02/part-1-from-chaos-to-clarity-how-to-structure-and-solve-problems-in-projects/","title":"Part 1: From Chaos to Clarity: How to Structure and Solve Problems in Projects"},{"content":"This post goes back to the beginning. In today\u0026rsquo;s world, where many companies are talking about AI tools and seeking to implement AI everywhere, I\u0026rsquo;ve noticed that something important is still missing.\nMost teams still struggle with basic project planning practices. What do I mean by this? Quite simply, teams often lack a clear methodology for planning their work. They create plans without looking at real data or past estimates (which often they don\u0026rsquo;t have that kind of data). They just keep adding more and more User Stories to Backlog, without a clear strategy.\nYou can probably guess what happens next: teams run around doing lots of work, but going ultimately nowhere. They build up technical debt. Sooner or later, this hurts the product quality and value for customers.\nSo here\u0026rsquo;s the problem: everyone is rushing toward AI, but they are forgetting that the basics still matter. Even AI tools won\u0026rsquo;t help if your basic planning process is broken.\nThink of it like running: You can\u0026rsquo;t just wake up one day and run 42 kilometres without proper preparation. Same with AI tools. Even the most amazing AI tools won\u0026rsquo;t fix bad processes for us - if we haven\u0026rsquo;t established solid foundations in planning and estimation.\nThat\u0026rsquo;s why I want to explain story points one more time - in the clearest way possible. I\u0026rsquo;ll briefly explain why they can be important, how they can help teams plan better, and how they improve teamwork.\nLet\u0026rsquo;s start with the fundamentals.\nWhat are Story Points are and How to Use Them In every project, teams face uncertainty: users\u0026rsquo; stories, tasks can be more complex than they appear, estimates can be inaccurate, and unexpected risks can have an impact on timelines. These challenges make planning and delivery a constant struggle.\nStory points turn subjective guesses into objective, consistent measures, providing projects with clarity, predictability, and a higher likelihood of on-time delivery.\nStory points are one of the most misunderstood and misused terms in Agile methodologies. People forgot that a story\u0026rsquo;s points should reflect relative estimation, experience, and knowledge rather than absolute time.\nThe Golden Rule: Story points are NOT a measure of hours or time required to complete a User Story. One story point does not equal one day.\nThe Three Pillars of Story Points To get an idea of how long something will take, a team should not just think about the time it takes.\nInstead, they should carefully examine the user story and consider three key factors:\nComplexity: How difficult is the task to solve? Does it require high-level logic or a simple update? Effort (Volume): How much work is actually involved? A simple task (low complexity) might still take a long time if there is a high volume of data to process. Risk and Uncertainty: Are there things we do not know about? Do we have the permissions, or can we use the Application Programming Interface? When the risk is high, the story point value is usually higher because we have to think about the delays, with the Risk and Uncertainty. Let’s illustrate this with an example:\nImagine two developers : one senior and one junior - both need to comparing a User Story from the Backlog.\nThe Senior Developer might estimate that the task will take 1 day. The Junior Developer might estimate that the same task will take 2 days. If we estimated purely in time, the same task would have two different numbers. However, the complexity of the task hasn\u0026rsquo;t changed. By using Story Points, both developers can agree that the task is, for example, a \u0026ldquo;3-point story\u0026rdquo; based on its difficulty, risk, and effort. This approach allows the team to plan based on the total amount of work they can complete, regardless of who does it.\nUpdating Story Points - When Things Change One important thing teams often forget: story points should be updated regularly. Change is constant in project management. Is the same with Story Points. When new requirements or information appear, technical challenges emerge - your estimates should be revisited. Story points are not “set in stone”; they should evolve as understanding improves. Ignoring this leads to unrealistic plans and frustrated teams.\nRe-estimating is not a failure, its a good practice. It helping teams:\nKeep plans realistic Learn from previous assumptions Improve future forecasting Because the foundation of Agile planning is adaptation, not sticking to an outdated estimate.\nHow to Record Your Estimates Once the team has agreed on a point value, it should be recorded in your project management tool (Azure DevOps, Jira (for Teams):\nAzure DevOps Adding Story Points to the Azure DevOps Board.\nJira (for Teams) Adding Story Points to Jira.\nAI and estimation Many companies try to apply AI everywhere, but story point estimation is one area where it is better not to rely on AI.\nOf course, AI can be a valuable - but it needs to be used wisely. When it comes to story points, AI should not replace people. Estimation is a human, team-based activity. Generative AI does not truly understand a team’s real capabilities. Yes, AI generated numbers can look “smart”, but this is not what we should expect from a good estimation. You don\u0026rsquo;t need false confidence for product owners and management. Automating story points doesn’t remove uncertainty; it often just hides it.\nSo, how can AI be used effectively in estimation? AI can support the estimation process without taking control of it by suggesting an initial range of story points based on historical patterns (if they exist), and then the team can discuss and adjust during refinement or planning.\nUsing an AI extension in Azure DevOps Board or in Jira in the right way can help highlight trends where the team has consistently over- or under-estimated, and also can support continuous improvement. Surface risks or anomalies by comparing new stories with similar past work. In this way, AI acts as an assistant, improving the quality of the conversation - while humans remain responsible for the final story point decisions.\nSummary Story Points can help teams estimate work based on complexity, effort, and uncertainty, rather than individual speed or assumptions. When used correctly, story points create realistic plans, improve predictability, and encourage conversations within the team.\nAI can support this process by highlighting patterns and risks, but can\u0026rsquo;t replace human judgment, team experience - story points work because teams are accountable for them, not because numbers exist in a tool. Strong fundamentals in estimation should come first - only then modern tools (including AI) can add true value.\n","date":"2026-01-19T10:03:09Z","image":"/uploads/2026/01/story-points.png","permalink":"/2026/01/the-importance-of-story-points/","title":"The importance of Story Points"},{"content":"In today’s remote-first world, trust is a product risk, and one that’s becoming harder to manage.\nBut what if some of your team members aren’t who they claim to be? That’s exactly what the DTEX DPRK cyber report uncovers.\nNorth Korean IT operatives are actively embedding themselves into global tech teams - using fake identities to fund weapons programs.\nThe DTEX report reveals how these actors operate. Your team could be targeted without knowing it. These actors embed as IT professionals and slowly build trust within the teams they work with. Their psychological tactic and patience have allowed them to infiltrate the global tech workforce deeply.\n🔑Here are some key takeaways Tech leaders and Project Managers shouldn\u0026rsquo;t afford to ignore 1. Remote Hiring Needs Rethinking For the HR team, it’s time to go beyond resumes and \u0026lsquo;gut feeling\u0026rsquo;. Video interviews, identity verification, and behavioural monitoring need to be part of the onboarding process, especially for distributed teams.\n• Verified identity checks.\n• Multi-step video interviews.\n• Device posture assessments.\n• Continuous behavioural monitoring once they\u0026rsquo;re onboarded.\nSecurity should start before someone is even hired.\n2. AI is fuelling both sides We’re excited about what AI can do - but so are they. Starting from generating fake resumes and headshots to automating phishing and identity spoofing, AI is making deception easier and more scalable by:\n• Synthetic headshots\n• Polished fake resumes\n• Automated phishing flows\n• Identity-spoofing at a scale we’ve never seen before\n3. DTEX shares clear red flags Watch for suspicious signals during interviews or early onboarding:\n➡️ Disabled screen locks\n➡️ Odd login times\n➡️ Excessive screenshot tools\n➡️ Hesitation during live coding or screen-sharing sessions\nRemote work isn’t the risk; unverified trust is - and security isn\u0026rsquo;t just an IT concern; it\u0026rsquo;s an operational one.\n","date":"2025-12-31T08:00:41Z","image":"/uploads/2025/12/cybersecurity-1.png","permalink":"/2025/12/what-if-your-remote-team-member-isnt-who-they-say-they-are/","title":"What if your remote team member isn’t who they say they are?"},{"content":"Managing Microsoft Fabric at scale quickly becomes painful if you rely only on the UI. Workspaces, capacities, and tenant-level settings all need repeatable, scriptable management. FabricTools is a community-driven PowerShell module that fills this gap by adding high‑level cmdlets focused on Microsoft Fabric and Power BI administration.​\nIn this post, you will learn what FabricTools is, how to install it from the PowerShell Gallery, and how to list all Fabric workspaces and export them to a CSV file for further analysis.​\nWhat is FabricTools? FabricTools is an open‑source PowerShell module that provides a set of helper cmdlets for administering Microsoft Fabric and Power BI. It helps with admins\u0026rsquo; daily workloads and DevOps / automation / SDLC scenarios, as well as any other task that is difficult to automate with only the UI or raw REST calls.​\nKey characteristics:\nCommunity‑driven and open source, hosted on GitHub under the dataplat organisation​ Distributed via the PowerShell Gallery as the FabricTools package​ Designed to complement existing modules (such as the Power BI admin cmdlets) by adding Fabric‑aware commands, not replace them​ Installing FabricTools from PowerShell Gallery Because FabricTools is published on the PowerShell Gallery, installation uses the standard Install-Module cmdlet. You can install it either for all users on a machine or just for your own profile.​\nInstall for all users Run PowerShell as Administrator and execute:\nInstall-Module -Name FabricTools If this is your first time using the PowerShell Gallery, you might be prompted to trust the repository; confirm to continue.​\nInstall for the current user only If you do not have admin rights on the machine or you prefer to keep everything scoped to your profile, use the CurrentUser scope:\nInstall-Module -Name FabricTools -Scope CurrentUser This places the module under your user profile directory, avoiding the need for elevated permissions.​\nInstalling a specific version In some environments, you may want to pin to a specific version of FabricTools (for example, to maintain consistency in CI/CD pipelines). The -RequiredVersion parameter lets you do this:​\nInstall-Module -Name FabricTools -Repository PSGallery -RequiredVersion 0.31.0 After installation, you can verify that the module is available:\nGet-Module FabricTools -ListAvailable If needed, import it explicitly:\nImport-Module FabricTools Authorising your PowerShell session with Fabric Before running any FabricTools commands, you need to authorise your PowerShell session against your Microsoft Fabric tenant using Connect-FabricAccount. This cmdlet handles the Azure sign‑in flow and retrieves an access token used by subsequent FabricTools calls.​\nBasic interactive sign‑in For a normal interactive session, you just pass the tenant ID and follow the browser/device login flow managed by Connect-AzAccount under the hood:​\n$TenantId = \u0026#39;00000000-0000-0000-0000-000000000000\u0026#39; Connect-FabricAccount -TenantId $TenantId -Verbose If you need to force a fresh login (for example, when switching users or clearing an old token), use the -Reset switch:​\nConnect-FabricAccount -TenantId $TenantId -Verbose -Reset The cmdlet stores the token and required headers in the internal FabricTools session object, so you do not need to deal with bearer tokens manually for later commands like Get-FabricWorkspace.​\nConnecting using a service principal (SPN) For automation scenarios (Azure DevOps, scheduled jobs, headless servers), authenticating with a Microsoft Entra ID service principal is usually preferred. You supply the tenant ID, the service principal (application/client) ID, and its secret to Connect-FabricAccount, converting the secret into a secure string first.​\nExample:\n# Tenant and SPN identifiers $TenantID = \u0026#39;02f0ae90-0000-0000-abed-209885e00000\u0026#39; $ServicePrincipalId = \u0026#39;4cbbe76e-0000-0000-93d9-1f75d371fde1\u0026#39; $ServicePrincipalSecret = \u0026#39;xxx\u0026#39; # store securely in practice # Secure way to connect with credential object $ServicePrincipalSecretSecure = $ServicePrincipalSecret | ConvertTo-SecureString -AsPlainText -Force Connect-FabricAccount ` -TenantId $TenantID ` -ServicePrincipalId $ServicePrincipalId ` -ServicePrincipalSecret $ServicePrincipalSecretSecure ` -Verbose ` -Reset This pattern allows your automation to run under a non-interactive identity with controlled permissions over Fabric resources, while keeping the secret out of plain text in memory. This is an example only, but remember not store any password in plain text. Load the secret from a secure store such as Azure Key Vault or a pipeline secret variable instead of hard‑coding it in the script.​\nListing Fabric workspaces with FabricTools Once FabricTools is installed and you are authenticated, you can use its workspace cmdlets to retrieve information about Microsoft Fabric workspaces. One of the core commands is typically exposed as Get-FabricWorkspace .​\nA simple pattern is:\nEnsure you are authenticated using the above step (Connect-FabricAccount). Call the FabricTools workspace cmdlet to retrieve workspaces. Export to CSV with Export-Csv. Here is a basic example:\n# Retrieve Fabric workspaces $workspaces = Get-FabricWorkspace # Export selected properties to CSV $workspaces | Export-Csv -Path \u0026#39;C:\\tmp\\FabricWorkspaces.csv\u0026#39; -NoTypeInformation What this script does:\nGet-FabricWorkspace returns a collection of workspace objects from your Fabric environment.​ Export-Csv writes the data to a CSV file that you can load (or just open) into Excel, Power BI, or a Fabric Lakehouse.​ You can use parameters when using Get-FabricWorkspace to find out a given workspace:\nWhere to go next FabricTools includes many more commands beyond listing workspaces, such as capacity operations, workspace creations, listing many other objects and other admin helpers.\nList of all supported commands is in the GitHub: FabricTools - Doc​\nIf you prefer, you can check the module’s help from PowerShell with:\nGet-Help -Name Get-FabricWorkspace -Full replacing the cmdlet name with whatever you want to explore.\nList of all cmdlets:\nGet-Command -Module FabricTools ​By combining FabricTools with standard PowerShell techniques (pipelines, filtering, exporting), you can build practical automation around Microsoft Fabric administration with relatively little code.​\nMore to come. Enjoy!\n","date":"2025-12-24T07:30:13Z","image":"/uploads/2025/12/2025-12-22-FabricTools-install-module-connect.jpg","permalink":"/2025/12/start-with-the-fabrictools-powershell-module/","title":"Start with the FabricTools PowerShell module"},{"content":"I reviewed the report on \u0026lsquo;AI Privacy Risks \u0026amp; Mitigations - Large Language Models (LLMs)\u0026rsquo;, and I realised something simple - we already have most of the tools to manage these risks. What we often miss is structure, ownership, and the habit of actually using them.\nIf you\u0026rsquo;re building anything with AI - even a simple chatbot or study tool -this also matters. These systems can leak personal data quietly or cross legal boundaries without anyone noticing.\nThe most common reasons why data slips through are: LLMs memorise small fragments of sensitive info. Users share personal stuff without thinking. Still not clear who\u0026rsquo;s responsible, the AI company or the people using it. Sometimes models are retrained on data they were never supposed to see. So instead of waiting for another regulation or checklist, maybe it’s time we get a bit more practical.\nHere\u0026rsquo;s how I see it: 1️⃣Know Where the Data Is Know where personal data moves - inputs, outputs, logs. Understand what your system sees before anything else.\n2️⃣ Focus on What Matters (Prioritise Risks) Focus on what could cause the biggest harm first. Not everything matters equally.\n3️⃣ Set Up Basic Protection Hide personal info before it goes to the model. Control who sees what. Filter AI outputs for sensitive stuff. Use \u0026lsquo;unlearning\u0026rsquo; tech where someone requests deletion. Keep humans in the loop on the critical decisions. 4️⃣ Test a Little Start with one simple, low-risk case. Monitor the alerts, learn from mistakes, and adapt quickly. The EDPB guidance emphasises a lifecycle-based risk assessment approach: before deployment, organisations should identify, classify, and evaluate privacy risks to understand potential impacts.\n5️⃣ Make Privacy a Continuous Process (Keep Doing It) Privacy shouldn’t be treated like a one-time setup; organisations should:\nRegularly reviewing and monitoring how data flows through the system to detect any privacy or compliance issues that may emerge over time. Adapt mitigation measures as needed: as usage evolves (new features, more users, different data types), risk profiles change - security, anonymisation, logging and oversight practices may need updates. Maintain documentation and transparency: even in this part, documentation is needed. You need to keep a risk register, log processing activities, document decisions, and ensure accountability. Thanks to these practices, the privacy in LLM adoption becomes not just a simple “checkbox list” but a living practice - part of corporate culture and technical operations. This long‑term approach can help organisations respond to new threats, regulatory changes, or evolving deployment scopes. Privacy shouldn\u0026rsquo;t slow AI down - it should guide us how we build it, driving us to create smarter, cleaner, and more transparent technology.\n","date":"2025-12-22T08:00:43Z","image":"/uploads/2025/12/Screenshot-2025-10-22-122352.png","permalink":"/2025/12/the-privacy-gap-in-ai-and-how-to-actually-close-it/","title":"The Privacy Gap in AI (And How to Actually Close It)"},{"content":"When companies decide to implement AI tools, the main argument is usually that AI will act as an employee assistant - helping people work faster and better.\nOn paper, it sounds like a perfect strategy. But in reality, this is often a first impression. Reason? Many companies don\u0026rsquo;t have their own AI strategy - they just copy ideas from popular sources. That makes AI implementations risky. Every company is a unique setup, and a one-size-fits-all approach usually skips over specific processes, the company culture, and what customers really need.\nSupport, don\u0026rsquo;t Swap☝ If leaders want to implement AI as a helpful solution for employees, they first need to understand where human and AI can work together - and where AI can\u0026rsquo;t help.\nLet\u0026rsquo;s be clear on this point: AI is not required everywhere. The smartest AI strategy focuses on knowing where it truly adds value.\nAI, of course, can handle repetitive and manual tasks really well. It can save employees hours, allowing them to focus on more complex, creative, problem-solving work. But still, even for automation, this fancy technology is not always the answer. Cheaper, simpler, existing solutions (email filters, calendar rules, or spreadsheet macros) often get the job done without needing fancy AI at all.\nEthics and Responsibility ✅ Leaders should also ensure that this technology fits with ethics, responsibility, and company values. As we all know, AI has no moral compass. In areas like healthcare or finance, a human loop is crucial.\nFor example, AI can analyse credit applications, highlight budget concerns, or shortlist job candidates, but humans must make the final call. You wouldn’t let AI approve a loan, sign a budget, or hire someone on its own.\nPartnership ⚖️ To collaborate effectively with AI, employees need ongoing training. They should learn how to use AI as a partner, not a competitor. Upskilling makes integration smoother, more effective, and ultimately unlocks the full potential of human-AI collaboration.\nTo do this effectively, organisations should regularly review AI-human collaborations:\n- How is this work?\n- Are the outputs accurate?\n- Checking if there are any blind spots in ethics.\nBuilding a culture where AI is treated like an employee assistant, not a team replacement, will help people see this technology as a collaboration, not a competition.\n","date":"2025-12-15T08:00:46Z","image":"/uploads/2025/12/Ai-and-humans.png","permalink":"/2025/12/humans-and-ai-how-to-find-the-right-balance/","title":"Humans and AI: How to find the right balance"},{"content":"Micromanagement kills teams☠️. It\u0026rsquo;s nothing wrong with checking what the team is doing or ensure they\u0026rsquo;re moving in the right direction - but managers, who constantly monitor every team move don\u0026rsquo;t build a strong team. These managers might think they\u0026rsquo;re doing a good job by keeping a close eye on everything, but in reality, it has the opposite effect.\nWhen teams feel their manager doesn\u0026rsquo;t trust them it impacts their motivation, engagement and performance. That\u0026rsquo;s why this kind of management is ineffective. Especially in today\u0026rsquo;s fast-changing environment, where every situation, challenge and customer is unique, teams should be empowered to think and decide for themselves.\nHow Does Ownership Help Build a Powerful Team?\nOwnership encourages the team to think outside the box, generate innovative ideas, and take responsibility when things go wrong. Fostering a sense of ownership can lead to several key benefits:\nTeams become more proactive, passionate, and motivated to create real impact for their users. Open communication leads to continuous learning and better collaboration. They can organise their work proactively. An Agile leader\u0026rsquo;s role is to foster an environment where individuals can grow, work together, build trust, and produce excellent work. The finest leaders cultivate a culture where people truly take pride in their efforts.\nFinding the Balance ⚖️\nGiving people ownership of their decisions needs to be done wisely. To do that, a Project Manager or Agile Lead needs to know when to step in and when to step back:\nIf they intervene too fast, teams may not be ready for this and as a result it will feel lost, flustered and confused - and this is not ownership. The same happens if the Agile lead does not take action or is too slow; the team can feel unsupported. It\u0026rsquo;s beneficial for an Agile Leader or PM to be a facilitator - actively supporting the team in achieving results. Their role involves helping the team grasp common goals and providing guidance when needed. Ownership cannot be forced. A good Agile Lead, PM, understands that fostering a culture of ownership isn’t about telling people what to do- it’s about guiding them to take responsibility and supporting them along the way. Key Questions to Encourage Ownership:\nWhat needs to be changed, added, or improved? How does my team typically grow? Do we need additional expertise, training, or mentoring? Are there blockers preventing ownership? How can the borders be aligned with the maturity? Summary:\nPeople need more than a salary to grow. They need a leader who provides a clear vision, someone who\u0026rsquo;s there when they need support, and a safe environment, where they can speak openly about challenges.\nFinally, remember: simply copying what worked for another team or company without understanding your team’s needs and goals won\u0026rsquo;t encourage true ownership.\n","date":"2025-12-08T07:23:52Z","image":"/uploads/2025/11/team-work.png","permalink":"/2025/12/want-to-build-a-strong-team-give-them-ownership-and-step-back/","title":"Want to Build a Strong Team? Give Them Ownership - and Step Back"},{"content":"More and more companies are diving into AI projects. From my perspective, many of these companies jumped into AI without a robust understanding of how the project process should unfold - not only from a data management perspective but also from a project management approach.\nAs a result, their AI project fails. This situation is quite similar to certain organisations adopting the Scrum framework simply because everyone else was doing it, without considering whether it truly suits their requirements.\nIn this post, I\u0026rsquo;ll share a project management methodology that I believe can be effective for AI projects.\n🤔 Why does Traditional PM Fail for AI Projects?\nAI project management is challenging - you can\u0026rsquo;t apply the traditional IT project management approaches, because, in a very short time, you will miss a core fundamental aspect of AI projects - data-driven, and that’s why the traditional project development life cycle does not work effectively for AI projects.\n🚨 The problem?\nCompanies taking AI projects are unfortunately not prioritising the understanding of client data and often employ traditional project management methodologies and frameworks. While this approach focuses on managing complex application development projects, it is not suitable for data-driven projects. Agile,e for instance, can\u0026rsquo;t be applied to AI projects solely because AI development is highly unpredictable and usually will not fit into a short sprint. Additionally, due to the nature of AI models, where advanced experimentation is necessary, the training cycle is longer and will not align with the Agile fast iteration cycle.\n💡 What can work?\nA successful AI project manager knows that data is the heart of AI - without it, machine learning systems cannot learn. However, having data alone is not enough; understanding data, preparing it, evaluating models, and finding the optimal way to implement data into AI systems are equally critical.\nThis is where Cognitive Project Management for AI (CPMAI) can help- complementing Agile methodologies, it offers a structured yet flexible approach to AI project lifecycles.\nThe CPMAI structures six phases to transform business needs into effective AI models:\n1️⃣ Phase: Business Understanding - “Mapping the business problem to the AI solution.”\n2️⃣ Phase: Data Understanding -“Getting a hold of the right data to address the problem.”\n3️⃣ Phase: Data Preparation -“Getting the data ready for use in a data-centric AI Project.”\n4️⃣ Phase: Model Development -“Producing an AI solution that addresses the business problem.”\n5️⃣ Phase: Model Evaluation - “Determining whether the AI solution meets the real-world and business needs.”\n6️⃣ Phase: Model Operationalisation - “Putting the AI solution to use in the real world, and iterating to continue its delivery of value”.\n💡Summary:\nUnderstand how AI is being applied Consider the nature of the project Understanding the data Successful AI projects require not only the right methodology but also a balance of structure and flexibility, putting data at the centre of every decision.\nCredits: Picture is from CPMAI-Methodology-Source-Cognitive\n","date":"2025-12-03T07:54:33Z","image":"/uploads/2025/11/CPMAI-Methodoogy-Source-Cognilytica.png","permalink":"/2025/12/how-to-manage-ai-projects-effectively/","title":"How to Manage AI Projects Effectively"},{"content":"According to a RAND report, private sector investment in AI grew 18 times more between 2013 and 2024, and over half of mid-sized companies have already deployed at least one AI model.\nManagers feel pressure to \u0026lsquo;do something with AI\u0026rsquo;, but most struggle to translate the leader\u0026rsquo;s ambition into action (AI transformation).\n[blockquote author=\u0026quot;\u0026quot;]AI adoption is almost everywhere today[/blockquote]\nThe biggest mistake? Organisations focus more on using cool technology than solving actual problems for their users.\nIf you\u0026rsquo;re solving the wrong thing, even the most advanced AI won\u0026rsquo;t help - it might make things worse 😐.\n🧠 First Understand the Problem: Don\u0026rsquo;t build a tool based on AI just because it\u0026rsquo;s trendy. Stop and think if AI is truly needed for this problem, because not every problem requires AI. Define the problem clearly. Like in each project, think: What user pain point are you trying to solve? 🔍 Use Research \u0026amp; Design Approaches: Ensure that technical staff understand the project purpose and domain context. Focus on data quality first - not having high-quality data available to train your model will give you ineffective AI models. Resist the \u0026lsquo;shiny object\u0026rsquo; syndrome - avoid bias toward the latest technology when simpler solutions work better. 🤖 AI Is Not One-Size-Fits-All: Choose the right AI model (if needed) - choosing the wrong tool could hurt your product. Avoid overengineering solutions. 🧩 Map Current Workflows: Before automating, understand the current user journey - how is the user currently trying to solve the problem? Which step can AI automate or improve? Consider what skills users need to address the larger problem. 🧪 Test Early with Prototypes: Use tools like Google AI Studio, or run a \u0026ldquo;Wizard of Oz\u0026rdquo; test. Validate assumptions with real users to uncover hidden problems. Think About Diverse User Experience: Different users face the same problem differently, due to language, location, and habits. Align AI behaviour with real user intentions. Make it natural, safe, and ethical. ⚡ Key Takeaways for AI Product Managers: Define the primary goal clearly - What should your AI system do for users? Consider sub-goals and alternatives. Prototype early Test AI concepts before building. AI can be a powerful tool in helping users solve problems. But before building an amazing AI solution, take a step back. Spend some time analysing the problem, understanding the users, and considering the context. That\u0026rsquo;s how you build AI that helps.\n","date":"2025-11-28T18:51:52Z","image":"/uploads/2025/11/Programming.png","permalink":"/2025/11/before-building-your-ai-solution-start-with-the-right-problem/","title":"Before Building Your AI Solution: Start with the RIGHT Problem"},{"content":"4 berth Swift Voyager 494A Trip: 3 weeks.\nPeople: 2+2\nCountries: 7\nDistance: 5,127.5 km (3,186 miles)\nLitres of fuel refuelled: 515.65 litres\nFuel consumption: 10.06 litres/100km OR 23.39 MPG (Miles Per Gallon)\nPeriod: 02/08/2025 - 21/08/2025\nTotal costs: £8,021 (inc. £4,184 motorhome rent)\n","date":"2025-10-26T19:47:54Z","image":"/uploads/2025/10/IMG_6480.jpg","permalink":"/2025/10/motorhome-trip/","title":"Motorhome Trip"},{"content":"🔍 The Challenge: DevOps for Microsoft Fabric In the world of Microsoft Fabric, DevOps is still maturing. Unlike Azure Data Factory (ADF), which has been around long enough to have established tooling - like the #ADFTools I developed 5 years ago - Fabric is new, broad, and complex, in a very positive way!\nMicrosoft Fabric integrates data engineering, warehousing, real-time analytics, and BI. With this scale, the need for solid DevOps tooling is more critical than ever.\n🛠️ The Spark: Why Another Tool? You might have seen my recent post discussing CI/CD readiness for Microsoft Fabric. That sparked further reflection. I began to seriously consider building a DevOps automation tool tailored for Fabric, just as I had done for ADF (Azure Data Factory).\nBut this time, something clicked: maybe I don’t have to start from scratch. That’s when I decided to take the lead and initiate a new community project - FabricTools - bringing together existing efforts under one unified, open-source PowerShell module for Microsoft Fabric.\n🤝 Collaboration Over Reinvention During last year\u0026rsquo;s Data Saturday Madrid (SQLSatMadrid), I spoke with Frank Geisler, who had already created a great PowerShell-based project called PowerRTI. Later on, I came across FabTools by Ioana Bouariu, another PowerShell module built about a year ago.\nRather than competing efforts, these tools presented an opportunity.\nSo, I reached out to both Frank and Ioana with a proposal:\nLet’s join forces, unify the code, and build a single community-owned module under the DataPlatform GitHub organisation.\nThey agreed.\n🔁 Stronger Together: Merging Efforts with Tiago Shortly after my first commit to the newly created project on GitHub, I spoke with Tiago Balabuch, a Microsoft employee and member of the Fabric ACE Team (a team helping enterprise customers adopt Fabric). Tiago had been working privately on a PowerShell module of his own: MicrosoftFabricMgmt\nHe had just made it public, and the timing couldn’t have been better. We started a deep conversation about what we both have done and what my vision of FabricTools. Tiago was very keen to join and support the project, which much more functions than we already have.\n🚀 That code has already been merged into the FabricTools project.\nI handled the integration myself right after creating the new project. We\u0026rsquo;ve unified much of the structure already, and more alignment and refinement are on the way. Tiago’s work brings a wide coverage of Fabric resources and a solid foundation that accelerates our progress significantly.\n🧑‍🔧 Support from the Best Before I started this new initiative, I had already looked at dbatools - a mature and very well-organised project being developed by the community. We wanted to bring that experience and values to FabricTools, so I spoke to Rob Sewell, co-owner of dbatools and administrator of the DataPlatform GitHub organisation. Rob brings deep experience in PowerShell tooling, DevOps, and community-driven open-source projects.\nBut that’s not all - Rob also invited Jess Pomfret, a well-known contributor in the PowerShell and SQL Server communities, to join the effort. Jess has deep expertise in PowerShell development and automation and is now part of the core team shaping FabricTools.\nTogether, Rob and Jess are helping us with:\nBuild and release automation Module structure and packaging Documentation planning (including a potential dedicated site) Long-term maintainability and governance And development and adoption of new features, too! Their involvement not only strengthens the technical side of the project but also reinforces our commitment to open, sustainable, and inclusive collaboration.\n🧱 Enter: FabricTools The result of that collaboration is FabricTools - an open-source, community-built PowerShell module for managing Microsoft Fabric.\n💻 Built by the community, for the community 🔓 100% open source 🌍 Hosted under the same GitHub org as dbatools 🚧 Currently in public preview 🧪 Early version – but designed for extensibility and future growth I\u0026rsquo;m inviting the community to test, contribute, and shape the direction of this project. You can already file issues and feature requests, and of course, contribute directly on GitHub.\n📢 Get Involved! This is just the beginning.\nWhether you\u0026rsquo;re using Fabric already or planning to adopt it soon, this module is your opportunity to shape the tooling around it. We encourage you to:\n⭐ Star the repo 🐛 Log issues ✍️ Join the discussions 🤝 Contribute code 📣 Spread the word As the project owner, I’m excited about the direction we\u0026rsquo;re heading - and even more excited to build it with you. Let’s make FabricTools the DevOps tool Microsoft Fabric truly deserves.\nWe\u0026rsquo;re building this tool together, and we’re just getting started.\n","date":"2025-06-20T11:56:56Z","image":"/uploads/2025/06/FabricTools-Baroque-oil-painting.png","permalink":"/2025/06/introducing-fabrictools-powershell-module/","title":"Introducing FabricTools (PowerShell module)"},{"content":"As I\u0026rsquo;ve observed how AI is transforming both my clients\u0026rsquo; and my own way of working, I\u0026rsquo;ve recently spent time exploring the topic of Vibe Coding.\nFrom the perspective of someone who manages projects and helps clients bring their ideas to life, ignoring Vibe coding would be a mistake. Of course, this method has its pros and cons, but let\u0026rsquo;s be honest – if something can improve and speed up certain processes for my work, I\u0026rsquo;m all for it😏.\nSince the best way to learn is by doing, I created a few simple apps using Vibe Coding. My goal wasn\u0026rsquo;t to launch the next unicorn start-up🚀, but to better understand what Vibe coding is, get familiar with the tools (I used Cursor), and see how this new way of working could support me in my daily tasks as a PM.\nHere are some things that helped me work with Vibe Coding more smoothly and effectively: Planning matters\nI\u0026rsquo;ve spent more time defining project requirements and writing a plan than actually building it. But honestly? Totally worth it. Having a clear vision made the whole process smoother, and it helped me know exactly where I was headed.\nBe careful with scope\nSince Vibe Coding makes creating things so simple, it\u0026rsquo;s easy to fall into the rabbit hole of adding more and more.\nPrompting matters\nThe clearer and more precise your prompt is, the smoother the process will be, and it helps you save tokens.\nFast feedback loop\nOne of the best things about Vibe Coding is how fast you can show a working draft to others. In just hours, you can bring an idea to life and get feedback.\nSecurity isn\u0026rsquo;t optional\nMy current apps are just for learning. In real projects, you need to plan for it from the start.\nTesting\nDon\u0026rsquo;t skip it! Frequent testing helped me catch issues early and see where my prompts still needed improvement.\nAI Assistant support\nOk, errors happened, my AI assistant was there to guide me through fixes( a great example of human + AI collaboration 😊).\nVersion Control\nUsing GitHub and committing regularly made it easy to roll back to a working version whenever I broke something 🙃\n💡 Takeaway ➡️ Client wants to see an idea? Show them in hours, not weeks.\n➡️ Need to test a concept? Build it, don\u0026rsquo;t just talk about it.\n➡️ Requirements change? (They always do) No problem - pivot fast.\nVibe Coding has shown me that embracing new tools and workflows doesn’t just speed things up - it also opens the door to smarter collaboration, faster learning, and more creative ways to bring ideas to life.\n","date":"2025-01-05T08:54:14Z","image":"/uploads/2025/12/vibe-coding.png","permalink":"/2025/01/how-vibe-coding-can-transform-project-management/","title":" How Vibe Coding can Transform Project Management"},{"content":"As someone who has spent over 20 years in the Software/Data Engineering area and has obtained numerous certifications, I can confidently say that certifications are still a valuable asset in today\u0026rsquo;s IT industry. Not only do they validate your knowledge and skills, but they also show your commitment to professional growth. Employers seeking competent candidates often prioritize those with relevant certifications. These credentials serve as concrete evidence of your dedication and proficiency, setting you apart from others competing for similar opportunities. So, if you\u0026rsquo;re wondering what the key to success in IT is, consider adding relevant certifications to your portfolio.\nWhat are your thoughts and experiences on this topic? Share them in the comments below.\nCertifications have long been a symbol of expertise and credibility in various industries. But in the fast-paced world of 2024, are certifications still relevant? Let\u0026rsquo;s uncover the power of certifications and why they remain essential today.\nIn a world filled with ever-evolving technologies and fierce competition, certifications stand as a testament to an individual\u0026rsquo;s knowledge and skills. They showcase a commitment to continuous learning and professional development.\nCertifications not only validate one\u0026rsquo;s expertise but also provide a competitive edge in the job market. Employers crave qualified professionals who can bring value to their organizations, and certifications serve as tangible proof of capability.\nFurthermore, certifications instill confidence in clients and customers. When seeking services or products, people often prefer certified professionals, as they offer a sense of trust and assurance in delivering high-quality results.\nThe importance of certifications extends beyond job opportunities. They provide a platform for networking and collaboration with like-minded professionals. Through certification programs, individuals can connect, exchange ideas, and stay updated on advancements in their respective fields.\nDespite the rapid growth of online learning platforms, certifications still hold their value. Online courses and educational resources may provide valuable knowledge, but certifications add credibility and recognition to one\u0026rsquo;s acquired skills. They highlight a commitment to going the extra mile.\nIn sectors where regulations and legalities play a significant role, certifications become even more crucial. These credentials ensure compliance with industry standards and protocols, safeguarding the integrity and safety of individuals and organizations alike.\nSo, whether you\u0026rsquo;re a professional looking to stand out in a competitive job market or a business seeking trustworthy partners, certifications remain an integral part of success in 2024 and beyond. Embrace the power of certifications and unlock a world of opportunities.\nDo you believe tech certifications are still a thing? If you\u0026rsquo;re not sure - watch the video and maybe I\u0026rsquo;ll convince you. If you agree - check this out, perhaps there are more reasons than you think.\nhttps://youtu.be/1vQLcQ_IS8I\n","date":"2024-02-23T11:00:29Z","image":"/uploads/2024/02/2024-02-22-Why-certifications-are-still-important-in-2024_-10-reasons.jpg","permalink":"/2024/02/10-reasons-why-it-certifications-are-still-important/","title":"10 reasons why IT certifications are still important"},{"content":"📰 News Power BI Project (PBIP) and Azure DevOps build pipelines for continuous integration\nIntegrating the PBIP format with Azure DevOps lets you use Azure Pipelines to automate CI/CD pipelines.\nUnity Catalog Permissions\nIn this article, Viralkumar walks us through implementation of Unity Catalog permissions model.\nArchitecting Global Data Collaboration with Delta Sharing\nDelta Sharing is Open, Flexible, and Cost-Efficient.\nImprove pull request descriptions using templates\nThis article shows you how to get started with pull request templates.\nUsing runMultiple To Orchastrate Notebook Execution in Microsoft Fabric\nNew great feature for Fabric to run multiple notebooks while shareing the compute and define dependency with DAG.\nWhat\u0026rsquo;s new and planned for Synapse Data Engineering in Microsoft Fabric\nRoadmap for Microsoft Fabric.\n30 Days to Learn It Select your own challenge - achieve your career goals.\n🎯 Shortcuts Create, configure, and use an environment in Microsoft Fabric\n📢 Upcoming Events SQLBits: 19-23 March 2024\nSQLDay: 13-15 May 2024\n🎬 Video DP-203: 24 - Selective CI/CD for Azure Data Factory with ADF Tools (part 1)\n😄 Smile Corner ","date":"2024-02-11T22:00:58Z","image":"/uploads/2022/01/last-weeks-reading-2023.jpg","permalink":"/2024/02/last-weeks-reading-2024-02-11/","title":"Last Weeks Reading (2024-02-11)"},{"content":"📰 News Fabric end-to-end use case: Data Engineering part 2 - Pipelines\nIf you\u0026rsquo;re new in Microsoft Fabric - check this series and build it with Sam.\nSet up Azure Monitor for your Python application\nUse OpenCensus Python SDK with Azure Monitor exporters to send different types of telemetry to Azure Monitor.\nDatabricks execution plans\nFalek helps us understand logical, physical plans and everything in between.\nQuery best practices\n\u0026hellip;in Kusto Query Language.\nPower BI Licensing Calculator\nStephanie Bruno created this calc to help people understand new options when Fabric appeared in Power BI landscape.\n🎯 Shortcuts Introduction to dataflows and self-service data prep\nHow to manage Sensitive Data in Cloud Analytics Platforms\nFabric Dataflows Gen2: To Stage Or Not To Stage?\nJSON Formatter\n📢 Upcoming Events SQLBits : 19-23 March 2024 (UK)\nSQLDay : 13-15 May 2024 (Poland)\n🎬 Video ARM Template Masterclass Episode 25: Bicep Modules\n😄 Smile Corner ","date":"2024-01-29T07:00:28Z","image":"/uploads/2022/01/last-weeks-reading-2023.jpg","permalink":"/2024/01/last-weeks-reading-2024-01-28/","title":"Last Weeks Reading (2024-01-28)"},{"content":"📰 News What happened in the DAX world in 2022\nNew DAX functions\u0026hellip; in 2022\nThe Microsoft Learn Cloud Skills Challenge\nMicrosoft Ignite edition is back!\nHow to build unit tests for Azure Data Factory\nHowever not much people are building it.\nACID Transaction in Fabric?\nWhat it does mean exactly for Microsoft Fabric?\nGetting hands on with Microsoft Fabric\nLet\u0026rsquo;s check what is not production-ready in Microsoft Fabric, by Paul Andrew.\nWhy Load Testing Power BI is Important\nPerformance is ok when only one developer testing it, then in Production\u0026hellip;\n🎯 Shortcuts Contoso Data Generator\nTales From the Cloud Data \u0026amp; Analytics Crypt #1 - Why this data transfer is so slow\u0026hellip;\nDelta Lake table optimization and V-Order\n📢 Upcoming Events SQLBits : 19-23 March 2024 (UK)\nSQLDay : 13-15 May 2024 (Poland)\n🎬 Video SQL Insider Series: Azure SQL DB - Automating scale up and down | Data Exposed 😄 Smile Corner ","date":"2023-12-18T09:43:53Z","image":"/uploads/2022/01/last-weeks-reading-2023.jpg","permalink":"/2023/12/last-weeks-reading-2023-12-17/","title":"Last Weeks Reading (2023-12-17)"},{"content":"📰 News Deploy a Delta Sharing Server on Azure\nWant to learn how to deploy a DeltaSharing server on Azure, but don\u0026rsquo;t know where to begin?\nWhy Structured Streaming and Delta Lake for Batch ETL?\nDenny Lee answers why use Structured Streaming and Delta Lake for batch ETL.\nSQL Projects support for Warehouse in Microsoft Fabric\nDatabase as a project in code repo. Always cool and best practice.\nUsing SSDT and DACPACs with Flyway: a Demonstration\nLearn how to use SSDT and DACPACs with Flyway, a tool for database migrations\nGitHub Repository Structure Best Practices\nThe post is a guide to GitHub repository structure best practices with a template repo and explanations of folders and files.\n🎯 Shortcuts Build a real-time dashboard by streaming events from Azure IoT Hub to Microsoft Fabric\n📢 Upcoming Events SQLBits : 19-23 March 2024 (UK)\nSQLDay : 13-15 May 2024 (Poland)\n🎬 Video https://youtu.be/At1IrqsM_sE\n😄 Smile Corner ","date":"2023-11-26T22:47:03Z","image":"/uploads/2022/01/last-weeks-reading-2023.jpg","permalink":"/2023/11/last-weeks-reading-2023-11-26/","title":"Last Weeks Reading (2023-11-26)"},{"content":"📰 News Process Azure Event Hubs data using Azure Data Factory Mapping Data Flows\nRead how to process Azure Event Hubs data using ADF for certain use cases.\nAutomate a pipeline migration to a Synapse workspace using Azure DevOps\nHelpful, when you\u0026rsquo;re planning to migrate from ADF.\nWhat\u0026rsquo;s the difference between Azure Synapse (formerly SQL DW) and Azure Synapse Analytics Workspace\nThis article helps people to not confuse these two.\nI made a new demo/workshop database\nDaniel provides an alternative to Stack Overflow database as a demo.\nIntroducing Microsoft Fabric: Data analytics for the era of AI\nWatch the new product from Microsoft stable.\nSQL Server Cursor Example\nA quick reminder how to use cursors in SQL Server databases.\n🎯 Shortcuts Cross-RDBMS Code Quality Reports in Flyway\nMicrosoft Industry Microsoft Fabric Content Hub by Erwin\nHow-To Deploy your Synapse Workspace Artifacts to a Managed VNET Synapse Workspace\n📢 Upcoming Events Microsoft Ignite : 15-16 Nov 2023\nSQLBits : 19-23 March 2024\nSQLDay : 13-15 May 2024\n🎬 Video 😄 Smile Corner ","date":"2023-11-05T07:44:35Z","image":"/uploads/2022/01/last-weeks-reading-2023.jpg","permalink":"/2023/11/last-weeks-reading-2023-11-05/","title":"Last Weeks Reading (2023-11-05)"},{"content":"Even now you can feel like a master of multitasking, and you don’t agree with me that multitasking doesn’t work. You can say: \u0026ldquo;I can do coffee and talk to someone, I can listen to music and walk, Fill out a simple form and listen to someone\u0026rdquo;, etc. But let’s be honest, these tasks don’t make impactful on your concentration because are simple and probably are part of a routine. The problem arises when the tasks are more complicated.\nIf you are doing more than one thing at a time you are worse and slower at all the tasks.\nI know what a lot of people say here, “ok, other people can’t multitask, but I can – I can talk by phone and drive a car, and I can do 3 projects at the same time”.\nBut the truth is different. If you are doing, let’s say, these three projects at the same time, you are distracted. You have trouble inheriting the impulse to do another activity and you can’t focus. Probably you say – “ok, I know how it works (or maybe how it is not working) but I can’t do one thing only. I have a lot to do. I have an executive role so I must do more things than others…”\nBut try to think in a different way. If multitasking will be ok, why do companies spend money doing research on the real effect of multitasking? The answer is simple: they realize that multitasking cost them a lot of money every year. One of that research examined 45 companies with an average of 20,500 employees and median annual revenue was $1 billion showing that, on average, multitasking caused these organizations to be 27.5 percent less productive than they could have been. A 27.5 percent loss in productivity from multitasking, with labor accounting for 32 percent of the total cost of projects, equates to a global loss of more than $450 billion a year. Those organizations came from a wide range of industries: engineering, software development, maintenance, and construction.\nIs still worth doing three projects at the same time by one person?\nWhy multitasking doesn’t work?\nIt’s not your personal interaction: it’s a biological human limit – sorry. Our brain was not created for multitasking. In a very short way – the brain can’t process two things (here I don’t mean routine, simple tasks) at the same time. Probably you realize that switching between tasks costs you time. We can concentrate on only one thing at a time.\nThe table below shows the waste caused by switching contexts:\nNumber of Simultaneous Projects Percent of Time Available per Project Loss to Context Switching 1 100% 0% 2 40% 20% 3 20% 40% 4 10% 60% 5 5% 75% (Source: Weinberg, Gerald M. (1992), Quality Software Management: System Thinking. Dorset House)\nThink: if you have ’only’ 3 projects, a 40% of your work goes nowhere. If multitasking workers don’t have everything, they need to finish a task, they may start working with incomplete information. This can cause them to stop before completing the task, which could block other team members.\nIn a classic work scenario, people try doing work everything at once. They believe it will be the fastest way to get things done. But the truth is the opposite of this. Switching between tasks, and project means that people do not do only half of their work but also, they are wasting their focus. When they want to back to where they were 10 minutes ago it takes them minutes just to get back to the same place where were before switching.\nFor many years companies believe that multitasking people can do work faster and they can control situations better. However, things play out a little differently.\nMultitasking people make more mistakes, have problems with concentration and are more aggressive and impatient. Multitasking increases stress. If you have a lot of things to control, you can hardly recognize bottleneck in your project.\nHow can a company prevent multitasking?\nIf you want to do effective work, and faster completion of the project you should stop multitasking. Before you start the next project, create tasks small enough for done in a real-time. Minimize the cost of switching between tasks – remember the rule: “half-done it’s not done”. Put tasks into blocks – in that way tasks are clearer and easier to organize. For planning and organizing you can use Azure DevOps Board, Jira, Trello, or MS Planner. During work, if you have possibly switched off your phone (I know this rule everyone knows but how many people follow this rule?)\nWorking on something important? The focus time method can help you. Focus time is two or more hours of completely uninterrupted work time for the most important tasks. The main goal of this method is eliminating outside distractions and concentrating on the work – during this time, there is no place for surfing on social media, talking by phone, or checking emails.\nAnother method that can help you achieve a project goal is to create a good project map (plan) with two main points: how to get from where you are now, to where you want to go. When you are planning this map, think about the eventual obstacles that you may encounter. Sometimes creating a good project map takes time but if you good define your project scope, and eventually obstacles it will help you move your project in the right direction.\nThe last, try not to take another task if you do not finish first – if you must take another task before finishing first, remember what you’ve done. It is easily back to work when you know where you finish.\nMultitasking it’s still a big problem for many companies. Most of them are still widely viewed as a requirement for employee soft skills. They believe that multitasking solves overdue projects, and speeds up work, but the truth is multitasking in projects is not only a waste of time, and money but also makes work worse and longer. The true secret of productivity is not doing a lot of things at a time is doing one thing at a time.\n","date":"2023-05-02T21:32:44Z","image":"/uploads/2023/02/multitasking-gc55751c1c_1920.png","permalink":"/2023/05/multitasking-chaos/","title":"Multitasking = chaos"},{"content":"If you\u0026rsquo;re experiencing delays and errors while working with SQL Server databases due to corrupt database files, you may feel frustrated and helpless.\nFortunately, Stellar Repair for MS SQL offers a comprehensive solution to these problems. This powerful tool help to recover lost/corrupt data from the primary (MDF) and secondary (NDF) database files of Microsoft SQL Server. It offers a user-friendly interface that makes it easy to use even for non-technical users.\nWhether you\u0026rsquo;re an individual user, an IT administrator, a data recovery consultant, or a database professional, Stellar Repair for MS SQL software has a three-license option that fits your needs.\nCorporate Edition: This edition Suitable for those businesses and organizations who need to repair corrupt MDF and NDF database files. With the Corporate Edition, you can easily recover all the database components such as tables, views, triggers, indexes, keys, and much more. Technician Edition: This is suitable for IT professionals, data recovery consultants, and database administrators who need to repair databases, reset passwords, and extract data from corrupted backups. In addition to the features offered in the Corporate Edition, the Technician Edition allows you to reset passwords of SQL Server accounts and extract data from damaged backup files. Toolkit Edition: This edition includes all the features of the Corporate and Technician Editions, including the ability to analyze SQL Server log table’s data and transaction logs. With the Toolkit Edition, you can view detailed information about all the transactions, happened on your database. Each edition comes with a one-year subscription including software updates and technical support. You can choose any one, based on the level of functionality and features required.\nSome best features of SQL Recovery software are:\nCompatibility: This software is compatible with all major versions of SQL Server, including SQL Server 2019, 2017, 2016, 2014, 2012, 2008, and all lower versions. Preview Option: The software provides a preview option for all recoverable objects. You can preview the data before saving it to ensure that you are recovering the right data. Recovery of Deleted Records: This software can recover deleted records from the MDF file, allowing you to recover lost data that you thought was gone forever. Recover Multiple Objects: With this software, you can recover multiple database objects like tables, views, triggers, keys, rules, and more in a single scan. Save Recovered Data in Multiple Formats: The software provides options to save recovered data in multiple file formats such as CSV, XLS, HTML, and SQL scripts. Simple and User-friendly Interface: The software comes with a user-friendly interface that makes it easy to use even for non-technical users. Efficient and Fast Recovery: The software has an advanced scanning engine that quickly scans and recovers data from the corrupt MDF file. Now, take a closer look at the specifications of the SQL Recovery software:\nStellar Repair for MS SQL is a database recovery software available on version 10.5.0.1. Multiple license editions are available, including Corporate, Technician, and Toolkit. The software supports the English language and was released in February 2023. The software provides a user-friendly and multiple-user interface with flexible license options. Stellar Repair for MS SQL requires an Intel-compatible processor (x86 or x64). The software requires a minimum 16 GB of memory, with 32 GB recommended for optimal performance. 250 MB of available hard disk space is required for installation files. The software is compatible with the following operating systems: Windows 11, 10, 8.1, 8, 7, Windows Server 2019, 2016, 2012, 2008, and 2003. Now, let’s have a look at the steps to use Stellar Repair for MS SQL:\n1. Discover or search for the corrupted SQL database MDF file.\n2. After selecting the SQL database MDF file, select Repair from the menu.\n3. Choose a suitable scan mode, and click the OK button to correct the file.\n4. The program displays the status of the database file repair operation.\n5. Click OK once the MDF file repair procedure has been successful.\n6. By choosing the Save option from the File menu, you can preview the recoverable database items and then save them.\n7. Choose the option to save the updated SQL database file, and then click the Next button to continue.\n8. Click the Next button after entering the information required to connect to the SQL Server and save the corrected MDF file.\n9. Select a suitable file-saving method, then click the Save button to continue\n10. The program provides status updates for the file-saving process, including Finished, Processing, Pending, and Aborted.\n11. A confirmation message appears following the successful completion of the saving process.\nConclusion In summary, SQL Recovery software by Stellar is a powerful tool that can help you to recover corrupt data from Microsoft SQL Server databases.\nWith its compatibility with all major SQL Server versions, user-friendly interface, advanced scanning engine, and multiple recovery options, this software is a must-have for database administrators, IT professionals, and anyone dealing with corrupt MDF files. Give it a try today and see the difference it can make in your work productivity!\n","date":"2023-03-15T00:17:34Z","image":"/uploads/2023/03/dan-cristian-padure-BxgVEo_rF-o-unsplash_700px.jpg","permalink":"/2023/03/stellar-repair-for-ms-sql-product-review/","title":"Stellar Repair for MS SQL – Product Review"},{"content":"Azure Synapse Analytics is not just \u0026ldquo;another service\u0026rdquo; in the Azure. It\u0026rsquo;s very comprehensive set of tools rather than one-goal-tool (like Azure Key Vault or even Storage Account).\nOn the documentation page we can read:\nAzure Synapse Analytics is a limitless analytics service that brings together data integration, enterprise data warehousing, and big data analytics. It gives you the freedom to query data on your terms, using either serverless or dedicated options—at scale. Azure Synapse brings these worlds together with a unified experience to ingest, explore, prepare, transform, manage, and serve data for immediate BI and machine learning needs.\nAs you can see, Microsoft uses word \u0026ldquo;worlds\u0026rdquo;. Those areas are as much big as worlds (thankfully not \u0026ldquo;universe\u0026rdquo;! \u0026hellip; yet). Simultaneously, Microsoft does all their best to make it user-friendly and easy for end-users or developers to achieve what they need.\nAnd all these things are good, until\u0026hellip; you will try to automate the deployment of the entire workspace (our universe - where all worlds belong to).\nTwitter Poll Like almost 3 years ago with ADF, I felt like there is still tons confusion of how to deploy Azure Synapse automatically and it\u0026rsquo;s about time to ask similar question to #AzureFamily. So I did and there is the result:\nAs you can see - I was right with my suspicious. People out there are either not decided yet what approach or tool to use nor they use vary approaches.\nAnd that is because the whole complexity of Azure Synapse and its deployment. Then you may ask:\nKamil, but why the deployment could be so much complex?\nWhy complex? If you follow me for a while, you might have seen my blog post about Two methods of deployment Azure Data Factory. That my journey started over 2 years ago and indeed some parts were difficult. Since then I prefer to deploy ADF from code (second approach) only.\nFor Synapse Analytics, there are even more various components belong to the ecosystem. Hence, it\u0026rsquo;s important to do it right. Right? Appropriately for the entire Data Platform and beyond. So, keep an eye on best practices and remember about limitations.\nWe are talking about a deployment here. Not manual deployment, but automated process. Ideally, we would like to build pretty decent and fast enough CI/CD process (potentially with Azure DevOps). Therefore, we should seriously consider a separation of two areas of Synapse Analytics workspace: Infrastructure - this will be deploy first, Application - this will be deploy as a second one assuming all infrastructure parts underneath are already deployed. The code of these two should be separated across separate folders in the same repository, or going further, you can split them even across separate repository (very likely under the same project).\nWhat do I mean here?\nFrom high-level perspective, I won\u0026rsquo;t be able to see\u0026hellip;\n\u0026hellip; and recognise some details as shown below:\nThere are two pillars: INFRA \u0026amp; APP.\nWhy does it worth to distinguish these two?\nVarious frequency of deployment (SDLC) between Infra vs App Separate team\u0026rsquo;s responsibility of Infrastructure part Security and costs control over Infra Faster development and more changes to the App repo/folder If you know any other good reasons for that - let me know in the comments section below.\nWorld #1: Infrastructure The following Synapse objects should be deploy as part of Infrastructure:\nWorkspace instance Integration runtimes Managed VNET + Endpoints GIT Integration Pools management: Dedicated SQL Pool Serverless SQL Pool (other than default) Spark Pool Credentials Synapse Access Control Deployment Solution 1) BICEP or ARM Template (Azure) BICEP is a domain-specific language (DSL) for deploying Azure resources. It provides a simpler, more intuitive syntax for describing Azure resources and their dependencies than the traditional Azure Resource Manager (ARM) templates. With BICEP, you can write declarative infrastructure as code (IaC) that is more concise, readable, and maintainable. BICEP files are compiled into ARM templates, which can be deployed using standard Azure deployment methods such as the Azure CLI, Azure PowerShell, and Azure DevOps.\nBICEP offers several advantages over ARM templates, including built-in functions, parameter validation, and better tooling support. BICEP makes it easier and faster to deploy and manage Azure resources, while reducing the risk of errors and simplifying the process of updating and maintaining your infrastructure.\nThen, when it comes to deploy Infrastructure part of Synapse Analytics - I recommend to use BICEP.\nNot sure how to build BICEP file and deploy Synapse with all satellite services? This GitHib repository is a deployment accelerator based on the reference architecture described in the Azure Architecture Center article Analytics end-to-end with Azure Synapse. This deployment accelerator aims to automate not only the deployment of the services covered by the reference architecture, but also to fully automate the configuration and permissions required for the services to work together.\n2) Terraform (Cloud agnostic) or\u0026hellip; if your cloud environment or client standard requires you to use Terraform.\nWorld #2: Orchestration (app) The first and the most important part of Synapse Analytics workspace, right after Infrastructure, is all orchestration objects:\nTriggers Pipelines Datasets Mapping Data Flow Linked Services SQL Scripts Notebooks Deployment Solution 1) azure.synapse.tools (PowerShell module) Disclaimer: I\u0026rsquo;m the author of the project. It\u0026rsquo;s been cloned from very successful ADF edition.\nI must admit this time Microsoft Synapse\u0026rsquo;s team got lesson learnt and brought the CI/CD tool for Synapse to the higher level.\nIn the same time, PowerShell for Synapse (Az.Synapse | PsGallery) have many gaps \u0026amp; inconsistency to be used for a deployment from code as I\u0026rsquo;ve done for ADF.\nHence, still in preview, so should be used in Dev or PoC environments only rather than in Production.\nFurthermore, due to complexity of Azure Synapse product and maturity of Microsoft\u0026rsquo;s DevOps extension (see below), I do not expect very intense development of this project in a long-term horizon. Keep reading.\nLink to GitHub repo: azure.synapse.tools\n2) Microsoft Synapse workspace deployment (DevOps extension) This time Microsoft did good job when it comes to the deployment of this Azure component. Similar task was not offered by Microsoft for Azure Data Factory. The extension to Azure DevOps contains only one task which offers 3 operation types:\nValidate - it validates the whole code from repo and generates ARM Template files. It\u0026rsquo;s only available in YAML version of pipeline. Deploy - publishes provided ARM Template to target Synapse workspace Validate and Deploy - in one-go action you\u0026rsquo;ll do both actions mentioned above (not recommended) What is not supported:\nResources, like pools management and workspace itself - see Infrastructure section above to deploy these things; Incremental deployment World #3: Serverless SQL Pool I\u0026rsquo;m not talking here about pool itself, where one will be deployed by default. If you need more serverless SQL pools you can always create them and then deploy - but that\u0026rsquo;s Infrastructure pillar (see above).\nIn here, I\u0026rsquo;m talking about all objects related to it, calls \u0026ldquo;metadata\u0026rdquo;:\nLake database External tables External views External file formats, etc There is no SSDT support for Serverless SQL Pool (yet), therefore we must build our own bespoke solution to create such objects by executing scripts incrementally. It can be done manually (not preferred option), or as a post-deployment step via PowerShell script or PySpark code.\nPotential Solutions Ha. None. Or almost none. There are some options but none of them are ideal and simple. Let\u0026rsquo;s deep dive:\n1) Redgate Flyway - does not work with Synapse My first, natural choice, was Flyway, the tool for migrating databases from one version to the another by executing incremental scripts. Unfortunately, when I tried to use it against Azure Synapse serverless SQL endpoints, it says: `Unknown SQL Server engine edition: 11`. I updated the ticket already created by other user in Flyway GitHub, but apparently there is no business justification for Redgate to introduce such update.\n2) DbOps (PowerShell) DBOps is a Powershell module that provides Continuous Integration/Continuous Deployment capabilities for SQL database deployments. The deployment functionality of the module is provided by DbUp .Net library, which has proven its flexibility and reliability during deployments.\nYou must remember to point out \u0026ldquo;nothing\u0026rdquo; for table monitoring versions:\nInstall-DBOScript (with -SchemaVersionTable $null)\nTherefore, all scripts deployed in such scenario should be idempotent and re-runnable. Not that much funny scenario\u0026hellip;\n3) SQL Server Data Tools (SSDT) If you are not familiar with SSDT - I have a few videos on my YouTube channel as well as comprehensive course on Learn with Azure Player platform.\nWait\u0026hellip; you said it\u0026rsquo;s not supported (yet). Yes, because this is what Microsoft says:\nHowever\u0026hellip; The very latest updates say that there are some options (features) available in SSDT to work with Synapse SQL Serverless. My MVP fellow, Kevin Chant, wrote about this last weeks some posts:\nDeploying a dacpac to a serverless SQL pool (15/02/2023) - in here Kevin explains how to extract (from physical instance of SQL) \u0026amp; deploy DACPAC to other instance using SQLPackage. Homemade serverless SQL Pool Database Project (28/02/2023) - this time there is a nice try of creating database project using dotnet, and afterwards: build it and deploy using SqlAzureDacpacDeployment task from Microsoft. Well, IMO it\u0026rsquo;s fair saying SSDT still doesn\u0026rsquo;t support Synapse SQL Serverless, at least for two reasons:\nThe edition is not available in Visual Studio External file format generates issues which suggest it\u0026rsquo;s not yet supported While we could live (for some time) with lack #1, the problem #2 is serious as in most cases we need SQL Serverless in Synapse to create External tables, which is not possible without External file format.\nBear in mind there will be new updates in this area, so I will be updating this post as soon as I spot something new.\n4) Custom solution with Python and Pandas for an incremental deployment Last year (2022) during DataRelay, in Birmingham I saw Arthur Steijn\u0026rsquo;s session who visited us for the first time from Rotterdam, The Netherlands. He presented his custom solution on project he was working on, and afterwards I asked him to share the code with me and he did even more creating blog post.\nWorld #4: Dedicated SQL Pool Dedicated SQL Pool is something previously known as \u0026ldquo;Azure SQL Data Warehouse\u0026rdquo;. Please do not confuse it with Data Warehouse you built in on-premise environments. This one is MPP architecture. Check out my session from SQLBits 2020 conference and/or the slide deck: \u0026ldquo;When and how to\u0026hellip; migrate to Azure Synapse?\u0026rdquo;. Having said that, we might have the following type of objects to be deployed:\nTables Views Stored Procedures \u0026amp; functions Security Deployment Solution Hopefully, there is SSDT edition for Dedicated SQL Pool in Azure Synapse.\nSSDT is SQL Server Data Tools, extension for Visual Studio that allows you to manage database project and deploy it incrementally by comparing project to the target server code. It\u0026rsquo;s very useful (state-based) approach available for Microsoft SQL databases (see supported target platforms on the picture aside).\nWorld #5: Power BI No additional efforts are required. All you need to do is Linking a Power BI workspace to a Synapse workspace, which actually is other way round, but\u0026rsquo;s ok. In this case, you\u0026rsquo;ll create new Linked Service and that\u0026rsquo;s all. Once your workspaces are linked, you can browse your Power BI datasets, edit/create new Power BI Reports from Synapse Studio.\nWrap up This is the first version of this article. I will update this post from time to time as many things is being changed around few areas described above here.\nOpen for discussion, question and suggestions.\n","date":"2023-03-07T07:44:43Z","image":"/uploads/2023/03/gears-1236578_1280.jpg","permalink":"/2023/03/synapse-analytics-workspace-deployment-challenges/","title":"Synapse Analytics workspace - deployment challenges"},{"content":"It was about time. The blog and all related services are being changed the name.\nFrom SQL Player to Azure Player.\nYou probably noticed over several years that the most popular conferences have changed their names:\nSQLGrillen to DataGrillen, SQL Relay to Data Relay. SQL Saturday stays the same, whereas similar Data Saturday appears SQLGla graduated to Data Scotland sqlserverdays to dataminds connect Fine, the biggest one in the UK - SQLBits still remains with the good, old one). The same for SQLDay in Poland. Don\u0026rsquo;t get me wrong. There is nothing wrong with SQL word in the name. I still like SQL Server and love SQL language. This is my background and where I started my professional journey over 20 years ago. But the Microsoft\u0026rsquo;s Data Platform evolves. Much. Very much. Nowadays it is tons more than just SQL Server or relational database. It\u0026rsquo;s entire bunch of services and applications: starting with fundamentals like Azure Storage accounts, Azure Data Factory, Azure Key Vault, Azure Synapse, Microsoft Power BI.\nAs you can see - Azure is everywhere now. Professionally for many years now, I work with Azure only, the work sometimes includes Azure SQL Database, but it\u0026rsquo;s about 10% of my work only.\nThat\u0026rsquo;s why my decision. And it\u0026rsquo;s not easy to conduct, because the blog is not the only one place where I contribute for community. There are several other places I do care of:\nGitHub, YouTube, Learning platform, Microsoft Marketplace, Twitter, Facebook. And, of course, I would like to keep all of them in sync and renamed. Therefore, the transition process just kicked off with the blog first and will follow other channels.\nDo fingers cross that the whole process goes smoothly.\nBut if you face any issue - please let me know via Twitter (DM).\n","date":"2023-01-01T19:26:02Z","image":"/uploads/2023/01/Azure_Player_Logo_Bitmap_RGB_950px_wb.png","permalink":"/2023/01/sqlplayer-is-renaming/","title":"SQLPlayer is renaming"},{"content":"Problem The first issue that developers in my team noticed was when they tried to create database with Spark:\nError message:\norg.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(meesage: Got exception: org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidAbfsRestOperationException Status code: -1 error code: null error message: InvalidaAbfsRestOperationException)\rat org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:112) This happens on the customer\u0026rsquo;s Synapse workspace where we have a public network disabled, so only private endpoint and managed VNET are available. Additionally, you probably spotted, that it took over 3 minutes to actually get this message. Hence, as a next step, in order to minimize the potential causes I simplified the query to make sure I have access to the Storage, by listing the files:\nThis still took over 90 seconds to complete, therefore, it looked like a networking/firewall issue. I started digging deeper and I thought that the network communication was blocked probably between Synapse Web UI and something that is called Apache Livy (REST API-based Spark job server).\nBTW. I checked all permissions to the storage (all of them, because we had more than one) - that was NOT the problem.\nSo, if I had appropriate permission to the Storage Account \u0026amp; container - I should be able to read some parquet files from there, right?\nSo, As you can see above - I could have read a parquet file using PySpark from the same Spark session (the same notebook).\nNext step we needed to double-check whether all DNS are being resolved correctly, hence being inside of the (private) network, we run such commands:\nnslookup web.azuresynapse.net\rnslookup storagename1.blob.core.windows.net\rnslookup defaultstorage.blob.core.windows.net Even from this perspective, everything looked good.\nRoot Cause Then, we started investigating Linked Services to the storages in the environment. We noticed that one of the Linked Services has not got Private endpoint associated:\nThen, very quickly we realized where the problem lied.\nThe issue actually was related to the fact we have 3 additional storage accounts, on top of those default ones come with Synapse.\nDue to private networking, we had created managed VNET + endpoints to the storage accounts\u0026hellip; except to the default one, which we were not planning to use as a storage.\nBut in such queries, I presented before, the access to the default storage is required and hence the error.\nSolution As always: when you know where is the pain - you probably can cure by fixing it. The same was in this case.\nOnce we created another (missing) endpoints to the \u0026ldquo;default\u0026rdquo; Storage Account and reconfigured Linked Service - everything started working.\nI hope this answer potentially saves someone else time in the future.\n","date":"2022-12-05T20:23:28Z","image":"/uploads/2022/12/InvalidAbfsRestOperationException_cover.jpg","permalink":"/2022/12/invalidabfsrestoperationexception-in-azure-synapse-notebook/","title":"InvalidAbfsRestOperationException in Azure Synapse notebook"},{"content":"Last weekend, I played a bit with Azure Synapse from a way of mounting Azure Data Lake Storage (ADLS) Gen2 in Synapse notebook within API in the Microsoft Spark Utilities (MSSparkUtils) package. I wanted to just do a simple test, hence I followed the documentation from Microsoft: How to use file mount/unmount API in Synapse.\nHaving an ADLS Account already created in a subscription - should be easy peasy, right?\nCurrently, there are three authentication methods supported:\nLinked Service, Account Key, SAS token. I used recommended method, which has access via Linked Service.\nProblem The first problem I faced when tried to run the first cell in a notebook (modified from doc):\nwith the following Trackback below:\n--------------------------------------------------------------------------- Py4JJavaError Traceback (most recent call last) \u0026lt;ipython-input-9-a8dbf6952053\u0026gt; in \u0026lt;module\u0026gt; 2 \u0026#34;abfss://stackoverflow@SQLPlayer2020.dfs.core.windows.net\u0026#34;, 3 \u0026#34;/mnt\u0026#34;, ----\u0026gt; 4 {\u0026#34;linkedService\u0026#34;:\u0026#34;LS_SQLPlayer2020\u0026#34;} 5 ) ~/cluster-env/env/lib/python3.6/site-packages/notebookutils/mssparkutils/fs.py in mount(source, mountPoint, extraConfigs) 37 38 def mount(source, mountPoint, extraConfigs={}): ---\u0026gt; 39 return fs.mount(source, mountPoint, extraConfigs) 40 41 def unmount(mountPoint, isLH=False): ~/cluster-env/env/lib/python3.6/site-packages/notebookutils/mssparkutils/handlers/fsHandler.py in mount(self, source, mountPoint, extraConfigs) 115 def mount(self, source, mountPoint, extraConfigs={}): 116 self.check_types([(source, string_types), (mountPoint, string_types), (extraConfigs, dict)]) --\u0026gt; 117 return self.fsutils.mount(source, mountPoint, extraConfigs) 118 119 def unmount(self, mountPoint, isLH=False): /opt/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args) 1255 answer = self.gateway_client.send_command(command) 1256 return_value = get_return_value( -\u0026gt; 1257 answer, self.gateway_client, self.target_id, self.name) 1258 1259 for temp_arg in temp_args: /opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py in deco(*a, **kw) 67 def deco(*a, **kw): 68 try: ---\u0026gt; 69 return f(*a, **kw) 70 except py4j.protocol.Py4JJavaError as e: 71 s = e.java_exception.toString() /opt/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name) 326 raise Py4JJavaError( 327 \u0026#34;An error occurred while calling {0}{1}{2}.\\n\u0026#34;. --\u0026gt; 328 format(target_id, \u0026#34;.\u0026#34;, name), value) 329 else: 330 raise Py4JError( Py4JJavaError: An error occurred while calling z:mssparkutils.fs.mount. : com.microsoft.spark.notebook.msutils.VerifyAzureFileSystemFailedException: [pre-verify before mount] request to https://SQLPlayer2020.dfs.core.windows.net/stackoverflow?directory=/\u0026amp;maxResults=1\u0026amp;recursive=false\u0026amp;resource=filesystem failed with exception - {\u0026#34;error\u0026#34;:{\u0026#34;code\u0026#34;:\u0026#34;OutOfRangeInput\u0026#34;,\u0026#34;message\u0026#34;:\u0026#34;One of the request inputs is out of range.\\nRequestId:8670ebf7-601f-00d6-5da2-9a7d0c000000\\nTime:2022-07-18T12:29:48.6231547Z\u0026#34;}}. at com.microsoft.spark.notebook.msutils.impl.mount.MSFsCommonMountUtils.verifyAzureFileSystem(MSFsCommonMountUtils.scala:153) at com.microsoft.spark.notebook.msutils.impl.mount.MSFsMountUtilsImpl.mount(MSFsMountUtilsImpl.scala:70) at com.microsoft.spark.notebook.msutils.impl.MSFsUtilsImpl._mount(MSFsUtilsImpl.scala:149) at com.microsoft.spark.notebook.msutils.impl.MSFsUtilsImpl.mount(MSFsUtilsImpl.scala:522) at mssparkutils.fs$.mount(fs.scala:37) at mssparkutils.fs.mount(fs.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) Caused by: com.microsoft.spark.notebook.msutils.VerifyAzureFileSystemFailedException: {\u0026#34;error\u0026#34;:{\u0026#34;code\u0026#34;:\u0026#34;OutOfRangeInput\u0026#34;,\u0026#34;message\u0026#34;:\u0026#34;One of the request inputs is out of range.\\nRequestId:8670ebf7-601f-00d6-5da2-9a7d0c000000\\nTime:2022-07-18T12:29:48.6231547Z\u0026#34;}} at com.microsoft.spark.notebook.msutils.impl.mount.MSFsCommonMountUtils.verifyAzureFileSystem(MSFsCommonMountUtils.scala:146) ... 16 more And because the 4th line was highlighted, I thought \u0026ldquo;ok, probably my Linked Service is wrong. Let\u0026rsquo;s check it out\u0026rdquo;. Then I opened my \u0026ldquo;Manage\u0026rdquo; hub, select \u0026ldquo;Linked Services\u0026rdquo; item in menu and clicked my LS_SQLPlayer2020 to edit it:\nThen, I click the \u0026ldquo;Test connection\u0026rdquo; button (right-bottom) and it says:\nGreat, Linked Service is configured correctly and works fine.\nInvestigation Let\u0026rsquo;s double-check the log then, which is not very helpful, but I focused on this part:\n\u0026ldquo;OutOfRangeInput\u0026rdquo; might suggest that a function tries to read some item which doesn\u0026rsquo;t exist in an array. That\u0026rsquo;s at least what my VB/C#/Python developer side of me whispered to me. In the first input parameter: \u0026ldquo;abfss://stackoverflow@SQLPlayer2020.dfs.core.windows.net\u0026rdquo; - the first word means container. Therefore, I double-checked if I really have \u0026ldquo;stackoverflow\u0026rdquo; container in the storage. I had.\nThen, after doing some research on the Internet and trying a few other things - I realised what was wrong.\nSolution That error message sometimes means you got either your account name or account key wrong. So if you attached with name and key or a connection string, please confirm you typed your account name right and that the key you are using is up to date.\nThe above answer comes from Matthew Rayermann (MSFT) who replied to a similar issue here on GitHub.\nTherefore, I revisited my code and change it to the following:\n1 2 3 4 5 mssparkutils.fs.mount( \u0026#34;abfss://stackoverflow@sqlplayer2020.dfs.core.windows.net\u0026#34;, \u0026#34;/mnt\u0026#34;, { \u0026#34;linkedService\u0026#34;:\u0026#34;LS_SQLPlayer2020\u0026#34; } ) Can you see the difference? In the second line - the name of the ADLS account must be lower-case (see: Resource name rules - Microsoft.Storage). I made this mistake by simply copy/paste part of the name from the Linked Service. When I resolved the issue, the command executed in 10 seconds, and then I was able to test a few other steps:\nThe goal of this test was to verify if my container is correctly mounted and has the right to read all folders.\nJobId Unfortunately, right now, mounted storage is available only from one job, which is related to the notebook that runs mount function. That\u0026rsquo;s something which is totally different from Databricks and I don\u0026rsquo;t understand why it\u0026rsquo;s been implemented in this way. Hopefully it will be changed by Microsoft team soon.\nAs always - thanks for reading! I hope you found this post useful and it saved your time.\n[spacer color=\u0026ldquo;264C84\u0026rdquo; icon=\u0026ldquo;fa-star\u0026rdquo;]\n[alert color=\u0026ldquo;green\u0026rdquo;]ℹ️ Azure Synapse ResourcesIf you want to learn more about Azure Synapse - check out this page on my blog where I collected the most interesting and useful links.\n[/alert]\n","date":"2022-07-18T14:04:06Z","image":"/uploads/2022/07/graphics-ga09b281dc_1200.jpg","permalink":"/2022/07/mounting-adls-point-using-spark-in-azure-synapse/","title":"Mounting ADLS point using Spark in Azure Synapse"},{"content":"Hello, data friends!\nLet me spread a few words about the new MVP led training initiative started by my friend and fellow MVP - Paul Andrew. It’s called Cloud Formations and seems to be positioned to follow in the footsteps of what ex-MVP, turned Microsoft Program Manager Chris Webb once did with Technitrain. For those old enough (like myself) to remember from Chris’ blog archive: https://blog.crossjoin.co.uk/category/technitrain/\nNow, the same expert-led training is available on various topics, currently running out of Birmingham UK, in person. Check out their website and events they currently have planned.\nI spoke with Paul and he was kind enough to invite me to present a training among the other fantastic speakers and friends, like Prathy Kamasani, Rob Sewell and Rishi Sapra. Although it\u0026rsquo;s very unlikely that will happen this year, keep an eye out for that. You can also use discount code STARTERFOR10 to get 10% off current tickets.\nThe value here for attendees is that MVPs generally aren’t trainers (which is a good thing), they are people like me passionate about technology and sharing that knowledge with the wider community. Paul could be on to something calling it a knowledge transfer session because that is what you get from an MVP, knowledge based on experience of using the technology first-hand. From the trenches if you like. Not just some theory translated into PowerPoint slides via Microsoft Docs. You get it first hand and from people that engage with the Microsoft engineering teams regularly - so they understand the direction of the technology landscape. A special type of trainer, maybe.\nI hope you can benefit from this initiative to progress your career and extend your skillset. I look forward to delivering my own knowledge transfer session in the future.\nThanks for reading.\nPS. Check out also my conversation with Paul, Rob or Chris, recorded already a few years ago. Do choose the audio or transcript version - whatever you prefer.\n","date":"2022-05-22T20:47:29Z","image":"/uploads/2022/05/Cloud-Formations-1400px-Logo-Card.jpg","permalink":"/2022/05/cloud-formations-a-new-mvp-led-training-initiative/","title":"Cloud Formations - A New MVP Led Training Initiative"},{"content":"Documenting objects dependencies of ETL processes is a tough task. Regardless it is SSIS, ADF, pipelines in Azure Synapse or other systems. The reasons for understanding the current solution can vary either: handover to other team/member of the team, troubleshooting, refactoring, debugging, investigating dependencies due to error, performance issue or others, as well as keen to remove selected/duplicated pipelines or logic.\nBut there is never a good time to make documentation, and even if that has been done - no one knows how much it\u0026rsquo;s up to date. The situation is not improved by the fact that quite often there is a lack of (free or built-in) tools for generating such documentation. Sounds familiar? I bet it does.\nWiki Markdown and Mermaid diagrams If you ever work with Markdown, you know that it is more and more popular format to develop and share documentation, often as a part of the code repository. Most popular tools support the format, including Azure DevOps and GitHub. Markdown supports diagrams in a few different forms, hence one can enhance documentation with a handful sequence diagram, Gantt charts or flowcharts, simply by writing a few lines of plain text.\nGenerate diagram with PowerShell It has been some time now since I released the first version of #adftools. The main reason behind the tool is to deploy Azure Data Factory directly from code. But as the name suggests - it is a toolkit. With the tool, you can very easily generate mermaid code and copy/paste it to your markdown file. What do we need to do?\nInstall/import PowerShell module (if you haven\u0026rsquo;t done that yet) azure.datafactory.tools - for Azure Data Factory azure.synapse.tools - for Azure Synapse pipelines Download or clone the code onto a local machine Run one line of code in PowerShell Copy the result and paste it into the target .md (markdown) file Let\u0026rsquo;s do all these steps together.\nInstall/import PowerShell module Depending on which code of which service you want to discover - install and import one of the following PS modules:\n$module = \u0026#39;azure.synapse.tools\u0026#39;\rInstall-Module $module -Scope CurrentUser\rImport-Module $module Download or clone the code If you don\u0026rsquo;t have Git Integration set up for the pipelines - do it. I was explaining how to do that in this post, and once you achieve that - you can clone the code into your local machine. The folder should contain the structure of subfolders similar to this one:\nRun one line of code in PowerShell It\u0026rsquo;s time to run \u0026ldquo;the magic\u0026rdquo;. Execute the following code (either in Visual Studio Code or PowerShell ISE / session):\n$RootFolder = \u0026#34;x:\\!WORK\\GitAz\\sqlplayer\\DataServices\\Synapse\u0026#34;\r$synapse = Import-SynapseFromFolder -RootFolder $RootFolder -SynapseWorkspaceName \u0026#39;whatever\u0026#39;\rGet-SynapseDocDiagram -synapse $synapse The first line is to set RootFolder variable with location of code. The second line (import) is going to read all files in subfolders. You\u0026rsquo;ll see similar result in output terminal:\nThe last line is to generate mermaid code for you:\nText representation of diagram Copy the result and paste it into the target .md (markdown) file Now, simply copy \u0026amp; paste the result into the existing or newly created MarkDown file (.md extension) and you should be able to see the diagram, whenever in Visual Studio Code or using Azure DevOps:\nPreview \u0026lsquo;diagram.md\u0026rsquo; file in Visual Studio Code Alternatively, you can pass the output of the Get-SynapseDocDiagram command directly to a new file:\nGet-SynapseDocDiagram -synapse $synapse | Set-Content -Path \u0026#39;synapse-diagram.md\u0026#39; Summary I hope the command will be useful for you. As always, should you have any comments or suggestions for improvements, let me know: Raise an issue.\nThanks for reading.\n","date":"2022-01-24T18:24:36Z","image":"/uploads/2022/01/adf_synapse_diagrams.png","permalink":"/2022/01/discovering-diagram-of-dependencies-in-synapse-analytics-and-adf-pipelines/","title":"Discovering diagram of dependencies in Synapse Analytics and ADF pipelines"},{"content":"Last Year (2021) I released a full and comprehensive course about SSDT: developing and maintenance SQL Server database within SQL Server Data Tools in Visual Studio.\nThank you to all those who decided to buy the course and congratulations to those who completed it. I know it was not easy, particularly from the fact the course is long. It covers probably all SSDT aspects, thus has 20 hours spread out on 64 lessons in 9 modules. I realised it may be too much.\nTherefore I decided to reorganize the lessons and modules of the course a bit so that they contained only the necessary knowledge about SSDT, and thus it was more affordable, in terms of size and price. As a result, I present his \u0026ldquo;little brother\u0026rdquo; named: Essentials of database projects with SSDT.\nIn order to be perfectly clear and transparent here - the comparison between these two editions can help you to understand the differences and decide which is better for you.\nEditions What\u0026rsquo;re the major differences between them? Let\u0026rsquo;s take a look.\nSSDT - Essentials Database projects with SSDT (Full) Price $99 ($14 per hour) $199 ($9.95 per hour) Modules 1+4 modules 1+8 modules Lessons / Length 37 / 7 hrs 82 / 20 hrs Satisfaction guaranteed * 15 days 30 days Slack - closed channel No Yes Quiz questions 20 \u0026gt;100 Available to buy All-time Only in specific periods * Satisfaction guaranteed means you have 15 or 30 days to ask for return full payment if (for ANY reasons!) you\u0026rsquo;re not satisfied with the course.\nAgenda This is exactly the same course, but the \u0026ldquo;Essentials\u0026rdquo; edition is mostly for beginners or for those who won\u0026rsquo;t learn advanced techniques and tricks. Having said that, let\u0026rsquo;s dig deeper and compare the agenda of these courses.\nDownload PDF\nWhat do they have in common? In both cases (editions) you get lifetime access to the courses and invoice at purchase.\nAny questions? Ping me: kamil@azureplayer.net\n","date":"2022-01-13T08:56:02Z","image":"/uploads/2022/01/Essential_SSDT_EN-blue.png","permalink":"/2022/01/database-projects-with-sql-server-data-tools-ssdt/","title":"Database projects with SQL Server Data Tools (SSDT)"},{"content":"? Press Efficient Upserts into Data Lakes with Databricks Delta\nWhen MERGE on data lake is inefficient.\nBuilding a Data Mesh Architecture in Azure – part 1\nWith this post, Paul began the series about Data Mesh in Azure. Is that for you or your project? Check it out.\nConfigure key auto-rotation in Azure Key Vault (preview)\nNow it can be done \u0026ldquo;automagically\u0026rdquo; for you, without additional efforts.\nAzure Data Factory CI-CD made simple: Building and deploying ARM templates with Azure DevOps YAML Pipelines\nDo automate your ADF deployment with YAML.\nAzure SQL Database and Managed Instance Import/Export service takes a long time to import or export a database\nWhatever you do, it\u0026rsquo;s just not working\u0026hellip; sorry.\nA (big) SQLPackage caveat\nIndexed views are something you should be aware of when doing migration within BACPAC file.\nYouTube Channel: Azure SQL\nI\u0026rsquo;ve added this link to the other listed here.\nResolve Git Merge Conflict for SSIS Projects\nHow branching in a git repository to work with multiple developers on the same SSIS project.\n?Upcoming events Festive Tech Calendar 2021\n1 Dec - 31 Dec 2021\nTechorama 2022\nMay 23-25\n? A word from Microsoft’s CEO ? Video Use Asserts for Data Quality and Data Validation in ADF\n? Smile Corner ","date":"2022-01-02T17:48:33Z","image":"/uploads/2019/11/news-tyler-nix-tPz-uoz5RBU-unsplash-1200.jpg","permalink":"/2022/01/last-week-reading-2022-01-02/","title":"Last Week Reading (2022-01-02)"},{"content":"? Press SynapseML: A simple, multilingual, and massively parallel machine learning library\nMicrosoft just announced the release of SynapseML (previously known as MMLSpark)\nSSH File Transfer Protocol (SFTP) support for Azure Blob Storage (preview)\nSFTP for BLOB Storage? Nice.\nThe best practices for organizing Synapse workspaces and lakehouses\nNot workspaces only, but also storage, folders and lakehouses\nService Endpoints and Private Link - What\u0026rsquo;s the difference?\nNetworking is not easy these days.\nPower BI Performance Tuning Cheatsheet!\nIntroducing SQL Server 2022: Top 3 New Features Announced at Microsoft Ignite\nHow Machine Learning has been used within the Music Industry\nIngestion and Processing Layers in Azure Data Lakehouse\nMicrosoft makes Visual Studio 2022 and .NET 6 generally available\nExecute a Fail activity in Azure Data Factory and Synapse Analytics\nPersonal Wi-Fi safety tips\n?Upcoming events Festive Tech Calendar 2021\n1 Dec - 31 Dec 2021\nTechorama 2022\nMay 23-25\nData \u0026amp; AI Summit 2021\nAll sessions and keynotes are available on demand.\n? A word from Microsoft’s CEO ? Video https://www.youtube.com/watch?v=sdtotHcbItA\n500-Level Guide to Career Internals: Building a Brand - Brent Ozar\n? Smile Corner ","date":"2021-11-28T23:50:12Z","image":"/uploads/2019/11/news-tyler-nix-tPz-uoz5RBU-unsplash-1200.jpg","permalink":"/2021/11/last-week-reading-2021-11-28/","title":"Last Week Reading (2021-11-28)"},{"content":"SQL Server 2022 is still in private preview (Microsoft MVP folks have access to it), but you can now read about the new release before it is released: SQL Server 2022 webpage: https://aka.ms/sqlserver2022 SQL Server 2022 blog: https://aka.ms/sqlserver2022blog SQL Server 2022 Microsoft Mechanics video: https://aka.ms/sqlserver2022mechanics SQL Server 2022 EAP registration form: https://aka.ms/EAPSignup SQL Server EAP Terms and Conditions: https://aka.ms/sqlserver2022eap_termsandconditions SQL Server 2022 datasheet: https://aka.ms/sqlserver2022datasheet Ignite BRK225 SQL Edge to Cloud Breakout session: https://myignite.microsoft.com/sessions/88fb4748-e5a9-4fc2-8ddb-68366473a8bd?source=sessions\n","date":"2021-11-22T12:14:42Z","image":"/uploads/2021/11/sql-server-2022.png","permalink":"/2021/11/sql-server-2022-is-coming/","title":"SQL Server 2022 is coming"},{"content":"When it comes to Azure Data Factory deployment, I always have a few words to say. You might be aware that there are two approaches in this terms. I was writing about them here: Two methods of deployment Azure Data Factory, comparing details. Hence, you may know that generally, I\u0026rsquo;m a fan of non-Microsoft, directly-from-code deployment approach and that\u0026rsquo;s why I built #adftools almost 2 years ago. To make our work simpler.\nSimultaneously, I understand why people cannot or prefer to use Microsoft\u0026rsquo;s approach using ARM Templates. Even though it still has some gotchas, but you know\u0026hellip; nothing is perfect. So, I\u0026rsquo;m fine with that and I even started thinking about how my #adftools can help also people who want to deploy ADF using ARM Templates. I\u0026rsquo;ve made steps towards this already adding a new cmdlet to azure.datafactory.tools (Publish-AdfV2UsingArm), which is just to deploy ADF from ARM using standard New-AzResourceGroupDeployment and all PowerShell samples from Microsoft Doc (there will be another post about it).\nExport ARM Template of ADF But, in order to deploy ARM Template files - firstly you must export an ADF instance to ARM Template format. You can do it manually from ADF Studio (Manage -\u0026gt; ARM Template -\u0026gt; Export) or automate this step by generating the file with help of Microsoft\u0026rsquo;s npm library. Because we like automation - we prefer to choose the latter way.\nMicrosoft ADF utilities It\u0026rsquo;s been about a year now since Microsoft published ADFUtilities npm package which enables new CI/CD flow for ADF.\nWhat changed? Let me quote the above documentation page:\nWe now have a build process that uses a DevOps build pipeline. The build pipeline uses the ADFUtilities NPM package, which will validate all the resources and generate the ARM templates. These templates can be single and linked. The build pipeline is responsible for validating Data Factory resources and generating the ARM template instead of the Data Factory UI (Publish button). The DevOps release definition will now consume this new build pipeline instead of the Git artifact. Great news, isn\u0026rsquo;t it? Finally, you can automate the build pipeline and consequently the entire CI/CD process, without using \u0026ldquo;adf_publish\u0026rdquo; branch.\nNPM package needs Internet connection Actually, it was weird to me when I was reading the documentation of the package for the first time. In order to export ARM Template files you should run:\nnpm run build export C:\\DataFactories\\DevDataFactory /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/testResourceGroup/providers/Microsoft.DataFactory/factories/DevDataFactory ArmTemplateOutput RootFolder (C:\\\u0026hellip;) - completely understandable. This is the location where all JSON files are held with all subfolders. FactoryId - WHAT? Why? Why the library needs information about SubscriptionId, Resource Group and Azure Data Factory name? I\u0026rsquo;m just asking to generate ARM Template for me based on the code I provided. It\u0026rsquo;s like a convert operation, both source ADF files and target ARM files are JSON format files. All required information is available locally. The documentation is not saying what\u0026rsquo;s those resources are: source ADF or target? In the first place, I thought that this is for target, but really\u0026hellip; both (source and target instance) doesn\u0026rsquo;t make sense!\nEventually, I figured it out.\nFactoryId should point to the original ADF. Why? Because the library is getting some information from adf.azure.com service via Internet! Yes, you must have an active Internet connection to be able to generate ARM Template correctly. This is insane. I noticed that this is NOT only in the situation when ADF uses Global Parameters (as I thought originally).\nDownloading \u0026ldquo;bundle\u0026rdquo; from adf.azure.com\u0026hellip; Therefore, it may fail in some scenarios. For some strange reason, the validation of Global Parameters used in pipelines happens via remote service rather than locally. If pointing ADF or its GP doesn\u0026rsquo;t exist you\u0026rsquo;ll get the following error message:\nFailure when the Global Parameter does not exist in live instance of ADF (sic!) Otherwise, it\u0026rsquo;s still connecting to adf.azure.com, and in a case of no Internet connection or access to the ADF service, you\u0026rsquo;ll not get ARM Templates as well:\nFailure when no Internet connection What\u0026rsquo;s all the shout about?\nIn most cases, you will be fine. Unless:\nYour CI/CD Agent has no access to the Internet (or outcome traffic is limited) OR Your original ADF has been deleted (or significantly modified) In the above cases, your CI (build) process fails and you won\u0026rsquo;t be able to carry on with the deployment.\nWhy is the process actually connecting to ADF? I do not know. Either way, it would be good if such information appeared on the documentation pages.\nBuild with NPM is 4 steps long Before you use the npm package, you must meet the prerequisites:\nInstall Node.js Install npm package Adding validation and export tasks, you\u0026rsquo;ll end up with 4 tasks just for ADF itself. It is a little bit\u0026hellip; disappointing. I would expect one step which does all the things.\n#adftools has these steps wrapped up in one cmdlet (PowerShell module) or as one task (step) for Azure DevOps:\nADF ARM Template Export task for Azure DevOps Factory-level features This part becomes harder and harder over time. Whenever the new feature appears in a factory-level object, it becomes a problem in the deployment process. What factory-level objects are? These are objects that define for entire factory and are kept in \u0026lsquo;factory/DataFactoryName.json\u0026rsquo; file if you have GIT integration configured, or\u0026hellip; these not available in the ARM Template at all (keep reading).\nThe first example is Global Parameters. The documentation says:\nThis information may be out of date as I saw the export command of npm package also covers Global Parameters and they are available in a created ARM Template file, however\u0026hellip; not always. I didn\u0026rsquo;t investigate this further, as usually I use my #adftools for deployment, so\u0026hellip; you know.\nAnother disadvantage related to Global Parameters is when you use \u0026ldquo;Private endpoint\u0026rdquo;. PowerShell script for deploying Global Params switches it back to \u0026ldquo;Public endpoint\u0026rdquo;. This of course would fail if you have Azure Policy which does not allow you to do so.\nIt has been raised by one of the users of my #adftools because I use the same method as Microsoft recommends:\nDeploying global parameters resets publicNetworkAccess setting on factory\nPublic Network Access = Disabled This property will never be part of the ARM Template as it\u0026rsquo;s an Infrastructure part of the service, as the Data Factory instance itself, so should be provided when creating a new instance of ADF. The problem is that Set-AzDataFactoryV2 cmdlet (the current version is 1.15) has no such parameter to support the property and there is no documentation at all on how to achieve that (ARM Template doesn\u0026rsquo;t contain this property as I mentioned earlier).\nVersion 0.x The npm package is still in version 0.x, which means \u0026ldquo;preview\u0026rdquo; to me.\nWhat\u0026rsquo;s the reason? Is that because the tool not ready for production environments? I wish to know as well as the roadmap of the utility: planned features or known issues.\nAlso, a simple Release Notes section would be nice to see somewhere in order to follow what\u0026rsquo;s changed. GitHub can really be helpful here ;)\nSynapse Readiness Even though Synapse Analytics have been GA\u0026rsquo;ed on 4 December 2020, a similar utility for Synapse pipelines is still not available:\nAutomated publishing is not available for Synapse Analytics Hence, there is no option to build an automatic CI/CD process to deploy pipelines, datasets, SQL scripts, notebooks and others to further environments. I\u0026rsquo;m glad that a few people already asked me if #adftools supports deployment to Synapse, but this is still not the case. Too many types of objects are unsupported by API, and who knows what else\u0026hellip; anyway, that would still be an unacceptable approach for many, so I truly believe that GA will eventually mean \u0026ldquo;Auto CI/CD features\u0026rdquo; are included and available.\nWhen it will be ready?\nSummary The goal of this post was particularly to point out significant shortcomings of building ADF with the NPM package from Microsoft as well as those less bothersome.\nPlease, do not take this as criticism, but rather as hints for everyone: for consumers - how to use and what well-known issues are, and for Microsoft - what should be fixed to make ADF users happy.\nPS. Due to the nature of the topic - this post can be updated in the future.\n","date":"2021-11-01T20:26:17Z","image":"/uploads/2021/11/people-3055939_1280.jpg","permalink":"/2021/11/skirmishes-with-automated-deployment-of-azure-data-factory/","title":"Skirmishes with automated deployment of Azure Data Factory"},{"content":"Today, I explain how to create a simple app in Microsoft Power Apps where:\nthe data are located in Excel, the table contains the path to the images from public Azure Blob Storage, the app displays images directly from Blob Storage based on the path defined in the database (Excel file). Why I used Azure Blob Storage to container my photo? Most people use Excel to create a table that can be imported to Power Apps. When we got the photos, it\u0026rsquo;s better to get them in Azure Blob Storage because it’s highly scalable and allocates the appropriate resources to keep up with demand. Furthermore, relational databases (like SQL Server) are not designed for storing binary data (like images).\nNow, a few words about my table:\nI created a table with fragrance. Inside the table, you can find the name of perfumes, brand, scent note, gender, and of course photos for every fragrance. Because I use an Excel table with the path to the images, I want to display these images further in my gallery control (in Power Apps).\nExcel File - Fragrances Table Before we go to the part where I create an application in Power Apps with Excel file as source, we must do the following steps:\nCreate a folder (container) in Azure Blob Storage and choose the type of container in Azure Blob Storage, where we gonna magazine our URL link with photos. Uploaded images to Azure Blob Storage Generate URL link how to copy a link with a photo from Azure Blob Storage to Excel and what to do after that. Step 1. Create a folder (container) in Azure Blob Storage and choose the type of container in Azure Blob Storage For the beginning - log on to portal.azure.com\nThen, search Storage accounts and create new storage (or use the existing one if you already have one).\nStorage Account in Azure Blob Storage Inside you can see four icons:\nContainers, File shares, Tables, Queries Types of storage in Azure Blob Storage for this case, I\u0026rsquo;ve opened Containers and on this level, I create availability for my link. After that, I added a new Container and named it - the name of a new container must be unique, just like the name of the file. Because every object which is stored here has an address that includes your unique account name.\nCreating container in Azure Blob Storage After that, I decided on the availability of the folder.\nYou have three availability options to choose from:\nPrivate - no anonymous access; Blob - anonymous read access for blobs only; Container - anonymous read access for containers and blobs. I selected Blob - this access level allows everyone to see my files, but not my containers. Which is perfect for this case.\nThree availability options. Step 2 Uploading images When a container was created successfully we can upload a file. Do click upload and open a new panel where we store a file.\nUpload file to Azure Blob Storage After that, you can download as many photos as you need and upload them to the container. Files can be uploaded to Azure Blob Storage in several ways, we use the easiest one - by Azure portal.\nSmall tips: When we opened the file you can see a list of all of my fragrance photos which I used to create my excel spreadsheet. All my photos got short-named. It\u0026rsquo;s easier to copy that name into the URL link. For this example, I use the filename \u0026ldquo;DylanBlue\u0026rdquo; (see picture below).\nAfter these steps, I’ve copied the file name and type of file [2] and paste them the into last part of my main URL link.\nFile with the name of photos. When the file was uploaded successfully, it\u0026rsquo;s available in the container - the picture below:\nUploaded photos to Azure Blob Storage. Step 3 Generate URL link From the list, I selected one photo (for this example: DylanBlue.jpg) [1], after that, I copy (to the clipboard) the URL link [2]. I\u0026rsquo;ve got the main URL link where I will replace the last part of the link with another name of photo, which will be used in the Excel table.\nWhen I paste the content of the clipboard to Notepad (or any other editor), I could see the whole link:\nGenerate a link from Azure Blob Storage. The link consists of three parts:\n1. the name of the storage account (sqlplayer2020 in this case)\n2. the name of our folder (container) - in this case: \u0026ldquo;publicp\u0026rdquo;\n3. the name of our photo and its format/extension\nThe last part will be changed each time - it contains the name of our photo and its extension. In this case, all files will be saved in JPG or PNG format.\nIn order to test whether the URL works fine - just paste the URL to the browser and an image should be shown:\nURL to DylanBlue.jpg file in Azure Storage Then you can see a photo assigned to the name:\nVersace Dylan Blue uploaded photo So, if we now know why we need Azure Blob Storage for storing photos, it will be great to show in simple steps how to use these photos in an Excel table.\nFilling up links to images in Excel file After I copied the main URL link to Azure Blob Storage, we gonna change only the last part of the link with the name of a photo, which we want to include in our table in Excel. The file must be in photo format (png or jpg), otherwise, Power Apps won\u0026rsquo;t be able to show it correctly.\nWhen we rename a URL, we should copy the link to Excel Table, one by one.\nTable in Excel Now, we have a table with all the perfume photos from Azure Blob Storage.\nBefore we import this spreadsheet to Power Apps where we will be creating our apps, we must convert [1] data (range of cells) into a Table with headers. Let\u0026rsquo;s name the table \u0026ldquo;Fragrance\u0026rdquo; [2]. PowerApps will be connecting through this name. The last thing we must do - is to save the file. I called it \u0026ldquo;Fragrance.xlsx\u0026rdquo;.\nConfigure table in Excel Meanwhile in Power Apps Last but not least, when finishing the design Excel table, we want to import the data to our Power Apps.\nWe log in to PowerApps: https://powerapps.microsoft.com/, where we will create a New App.\nAfter login, I choose Canvas Apps from Blank:\nBlank Canvas - Power Apps After this, I’ve named my new app – Fragrances and I choose the Tablet layout.\nWhen Power Apps opened an environment we can see a simple interface where we can design a new app.\nCanvas interface in Power Apps Left pane: Tree View - presents a hierarchical view of all the controls on each screen; Right pane: includes layout, properties, data sources; Middle pane: designing page, showing your canvas app; Before we started creating a new app, we must load data into Power Apps. On the Tree View click Data [pane 1], after that Add Data [picture below - number 2] - in this example search OneDrive for Business, because I used this service to store my Excel file.\nAdding OneDrive for Business data source. When you click on the connection item on the list, in the next step, Onedrive will return all Excel files stored there. Find and choose the preferred Excel file.\nChoose the Excel file with the table You will see all tables in the selected Excel file - only what you should do here is to select the appropriate table to be added to your Power Apps and click Connect. At the moment Excel data source is available in your app.\nChoosing the table from an Excel file Your Excel table is ready to use as a data source in Power Apps. (Now we need to configure components from Excel file with our app and photo.)\nThe connection is ready to use When the table is added to Power Apps we must insert the gallery component from the layout section to display the data from the table.\nYou can change the layout at any time by selecting value from Layout property, as shown on the picture below:\nWhen you choose a preferable gallery it will look like this on your canvas. The application encourages you to select a data source because only in that way this component makes sense.\nAfter that, when the layout is applied, we can edit Fields. Fields in this case are equivalent to the column in the table, they are represented by a row in the table and contain a data item for each row. Fields can contain different data types (Whole Number, Time, Data), Multiline Text. You can change the order of the fields - select one field and move it up or down.\nEdit fields - changing their orders Now Fields and styles are applied. In the first place, I had a perfume brand name, perfume name, and which fragrance group it belongs to.\nConfigure picture to be shown In this section, we configure Items on the list of Apps Gallery to display photos from Blob Storage. Select the component and click edit icon [1] to be able to configure an Item in the gallery. Then click on one of the photos [2] and on the formula bar select the Image instead of Items, and write this formula: ThisItem.Images.\nAll photos should automatically be updated in the app.\nPhotos displayed in the gallery component For better loading files with images be sure:\nthat the total of your photos don\u0026rsquo;t exceed 200MB any files images can\u0026rsquo;t be bigger than 64 MB Power Apps supports the following photo types: .jpg, .png, .gif Now we only need to save this app, publish or share it.\nThat\u0026rsquo;s it. In a simple way, we added Excel Table to Microsoft PowerApps, where the photos were located in Azure Blob Storage. Azure Blob Storage is widely used to keep your binary data.\n","date":"2021-09-27T13:12:22Z","image":"/uploads/2021/09/cms-265126_1920.jpg","permalink":"/2021/09/showing-images-in-gallery-using-url-to-blob-storage/","title":"Showing images in gallery using URL to blob storage"},{"content":"Man does not live by work alone. Everyone needs a hobby, preferably in isolation from their daily work. There is not much to read in this post. As a teenager, I was a fan of Batman, bought comics regularly, and was interested in the whole story surrounding the character.\nThe 1989 movie \u0026ldquo;Batman\u0026rdquo; was the first movie in which I was in the cinema alone, without my parents. I was 11 at the time.\nI still remember the place - it was the \u0026ldquo;Warszawa\u0026rdquo; cinema \u0026hellip; in Wrocław (Poland).\nI did not think that at this age I would be playing with Lego bricks. In addition, thematically related to Batman. I recorded several days of assembling Batmobile with Lego bricks and here you can watch the video with a very curled-up pace with only one appropriate soundtrack.\nEnjoy! Necessarily with sound. (\u0026lt;3 min)\n","date":"2021-06-25T10:33:39Z","image":"/uploads/2021/06/TitleScreen.jpg","permalink":"/2021/06/hobby-original-batmobile-1989-from-lego/","title":"Hobby: Original Batmobile (1989) from Lego"},{"content":"Components are reusable building blocks for canvas apps so that app makers can create custom controls to use inside an app. When developers start from the blank canvas they can create components that can define the characteristics of the app\u0026rsquo;s screen elements. A component can be a form, a button, a text box, they can become reusable on multiple screens. Components in Power Apps function consist of multiple attributes that users can apply in one step.\nIn this post, I will create a simple component. I will build a text header that we can later pin to other screens and uses in different applications. It will be a local component. Local means that I have to create w single components in a single app. If I was inserted this local component into different apps, I would have only a copy of this component. Component libraries are the recommended way to reuse components across the apps. When using a component library, an app maintains dependencies on the components it uses. To use the components from the component library, you must create an app that uses the component library.\nI think it\u0026rsquo;s good for the beginning defined features of good components. Because the attribute of a good component in PowerApps is not only that it exists, but should certain some functions.\nAttributes of a good component:\npossibility to pin it to new screens, using it in new applications with a similar control panel, should adapt in size to each new screen, can be reused by changing the attribute, the background colour, size, ability to attach some photo etc., the ability to attach a photo. can be added to a new custom property. Components have also limitations. Below you find the limitations components list. All this list you can find on Microsoft\u0026rsquo;s doc site: Known limitation.\nLimitations of components:\nYou can\u0026rsquo;t save data sources, forms, and data tables with components. Collections in components are not supported. You can\u0026rsquo;t insert a component into a gallery or a form. A master instance of a component is a local master and scoped to the app. If you change a master instance, only copies of the component within the app will reflect the change. Copies in other apps will remain the same unless you import the component library again. All master instances in those apps will be automatically detected and updated. You can\u0026rsquo;t package media files when you import a component. Components don\u0026rsquo;t support the Update context function, but you can create and update variables in a component by using the Set function. The scope of these variables is limited to the component, but you can access them from outside the component through custom output properties. How to start This is the part of the post where we start creating a new component\nIn the beginning, I’ve created a new application in PowerApps. I will pick the tablet layout component. Remember when you once choose what kind o layout you want you will not have the possibility of changing them during creating a component, also you can only use this component in other apps about the same layout.\nCanvas app from a blank When our application is open, and we named it, you can see on the Tree view two things: screens and components. By clicking Components you can create a new component, which will be accessible to any screen in the app.\nTree View We choose Components (1) and click**+** which means you want to create a new component (2). After click on this, you will see a new blank canvas - here we will create a component. Don’t forget to name your component. I’ve named mine: Header.\nCreate a new component Tips: Probably if you have existing applications you can’t see this component on the left side - you should turn on your components. To do this you go to the settings (1) -\u0026gt; advance setting(2) and there find components (3) and turn them on (4).\nWhen we have the first steps behind us, we can build the component now.\nAfter these steps, I’ve clicked in insert label and drag this across the size. After that, our components will be put on the canvas - picture (2).\nStep to insert label. Put the components on the canvas. Next, I go to Home on the ribbon and there I can edit my component a little - set the inscription in the middle, add a colour, etc.\nHome - ribbon After this, the component looks just like that - I\u0026rsquo;ve changed the colours, size, and position of the text - just like in (Microsoft) Word, you can change the attributes of your components on the ribbon.\nEdit component Just like you can see, my header has got high and width. Because we chose the canvas option for the tablet, our components dimensions will have a suitable size for a tablet.\nComponent Width Component Height. There we go\u0026hellip; we add the components right to the screen. Just like you see the component not fit as well, it has its own high and width. If we create the component we want to insert them into the screen. In this case, we should back go to the screen on the Tree View, and there choose Insert after click custom. You see there is a list. You can import them or export them to other apps – in the first place will be a component that you created. Now, only what you should do is click on them.\nAdd components to the screen There we go\u0026hellip; we added the components right to the screen. The component is not fit as well - I can drag them around, but it doesn\u0026rsquo;t fit the size of our screen and it has its own high and width. So, now I have to adjust the dimensions of this component then it will fit my screen. To do this I\u0026rsquo;ve backed to the component from the screen and use an object in Powers Apps called App.ActiveScreen. Apps.ActiveScreen means a component that \u0026ldquo;knows\u0026rdquo; the screen where the current component will be placed. In the first step, I go to the label and enter the formula App.ActiveScreen.Width at the moment my components will automatically adjust to the screen width.\nApp.ActiveScreen.Width When setting the height of the component, I also enter the same formula, with the difference that in the object I mention Height and in the formula, I only replace the word: App.ActiveScreen.Height\nApp.ActiveScree.Height I can add another screen in the same application but my header should be got the same colour and size just like my master components. To do this I go to label and enter the formula:\nParent.Width,\nParent.Height,\nSo that my components will automatically adjust to the width of the screen, and that every change contained in it appears on the other screens. When we apply this formula and when we go back to our screen, we will see that we can move our header to the right width and it will fit perfectly with another screen. This component can be reused on other screens in my apps. For example, if I\u0026rsquo;ve changed the colour background in my main chooses component on the other screen colour components will be changed.\nAdjust component to the screen. To add this component to other screens in the screen click a Home -\u0026gt; New screen -\u0026gt; choose blank screen\nAfter this step, you will create a new blank screen where you can add your components to this screen.\nInsert component to a new screen. Just like you can see, components have te same size, colour, height as the main component.\nAdjust the size colour component. Now, we know how to change the width, the height of our components. So in another part of this post, I will describe how to add this component to custom properties.\nCustom properties Custom properties are the advance customizations provided by the PowerApps. A component can receive input values and emit data if you create one or more custom properties.\nWe have two types of custom properties:\nInput property is how a component receives data to be used in the component. Input properties appear in the Properties tab of the right-hand pane if an instance of the component is selected.\nOutput data is usual for emitting data. It will take out from the component and gave it back to called screen. When you create an output property, you can determine what other controls can refer to the component state.\nCustom properties are located on the right panel, they appear when we click on our components. After clicking on it, you will see a panel on the right with options to create properties:\nNew custom property When we click + New custom property here you can create input properties and output.\nAdd a new custom property. You can choose a specific type of data that will be selected for a specific component. In this scenario, I choose Input and data type Text - depending on your needs, you can choose a different type of data.\nType property After this step, we can see a new property in the right corner.\nRemember to name properties and in the Name window so that there are no spaces when entering the name.\nThen the name of the properties will appear at the bottom of the right pane. To add this new property to our label, we need to change to components before Text - enter the name of our components, then go to the label and change the title of our components there. I must enter the following formula; Header.Header, you can see that the name of the new components shows our custom properties.\nI\u0026rsquo;ve changed the default header:\nBecause my label without the components is hardcoded, I want to put it into the property. To do that I must write the name of my components and name my property. After this, you will see that component change the name everywhere from Text on Header.\nThis way all remaining screens will have our header text. We can change the name of our component by typing in the right panel a new name for each screen.\nThis component can be reused on other screens in my apps. For example, if I\u0026rsquo;ve changed the colour background in my main chooses component on the other screen colour components will be changed.\nComponents in Power Apps can consist of multiple attributes that users can apply in one step. Components can simplify your work with Power Apps. Thanks to them, we can create characteristics buttons, shape, colours. We can reuse them in other apps, which help us to save our time and unifies our apps.\n","date":"2021-03-12T07:00:55Z","image":"/uploads/2021/03/Image-by-Gerd-Altmann-from-Pixabay-.jpg","permalink":"/2021/03/canvas-apps-components/","title":"Canvas Apps Components"},{"content":"Both, Azure Data Factory and Azure Databricks offer transformations at scale when it comes to ELT processing. On top of that, ADF allows you to orchestrate the whole solution in an easy way. In a case, when you prefer to use Scala, Python or SQL code in your process, rather than Mapping Data Flow in ADF - you must link ADF to Databricks. I will show you how to do that in two ways.\nIn this short video, I will present two methods and their differences, and as always the demo:\nDEMO 1: Connect via PAT (Personal Access Token)\nDEMO 2: Connect via MSI (Managed Identity Authentication)\n","date":"2021-02-04T22:23:20Z","image":"/uploads/2021/02/ADF-with-Databricks-set-up-connection.jpg","permalink":"/2021/02/set-up-connection-from-azure-data-factory-to-databricks/","title":"Set up connection from Azure Data Factory to Databricks"},{"content":"Azure Data Factory is a fantastic tool which allows you to orchestrate ETL/ELT processes at scale. This post is NOT about what Azure Data Factory is, neither how to use, build and manage pipelines, datasets, linked services and other objects in ADF. This post is completely focused on the topic of deployment (aka publish) Azure Data Factory from code to the instance (service) in the cloud.\nDeveloping pipelines in ADF I\u0026rsquo;ve been working with ADF since 2018. At the time, the whole ADFv2 was in preview and the product was (and still is) fastly develop by MS Team (kudos to Mark Kromer and his team). Today, ADF is a pretty mature product and offers a lot of useful features, including Global Parameters, Mapping Data Flow, GIT integration and much more. We are focusing here on governance the code of ADF and how to deploy it from a code repository to the service, hence mentioning the integration with GIT was on purpose.\nWhile working with ADF UI by the browser may not be a preferable way for many (there is no Visual Studio support for ADFv2 as it was for v1 and Microsoft is not planning to do so in nearest time), the UI is better and better and doesn\u0026rsquo;t cause much trouble when building pipelines. Also, as long as you develop the solution only in one instance (let\u0026rsquo;s name it DEV environment), you will not spot any issue or problem. Until you try to promote and deploy the code of ADF to further environment (or stage - using naming nomenclature from Azure DevOps).\nTwo approaches to deploying ADF I used ADF in a few organisations, so far, for the last few years.\nIn most cases, we used an approach of deploying ADF directly from code, from a branch in GIT repository. I knew that Microsoft recommends only one way of CI/CD for Data Factory. The whole concept and details are described here: Continuous integration and delivery in Azure Data Factory.\nUnfortunately, I found this way of little use and hard to use. Hence, then I realised: hey, if I feel like that and used a different publish approach a few times - maybe the others also have similar problems or just prefer his own approach of doing this? I did a quick poll on Twitter asking people what kind of method they use:\nTwitter Poll: Who uses what? Turned out that two-third of people use Microsoft\u0026rsquo;s deployment way, according to their answers on that poll (including few people who publish the code manually). In 1/3 cases people prefer to deploy directly from code. You may ask: what\u0026rsquo;re the differences? What characterizes both methods? Which one is better?\nBefore I start answering these questions, let me present both methods of publishing.\n#1 Microsoft approach (ARM template) In both approaches, you must set up GIT integration to assign your ADF service to the selected repository. If you are not sure how to achieve that - I described it here: Setting up Code Repository for Azure Data Factory v2. As a developer, you can work with your own branch and can switch ADF between multiple branches (including master/main). How this is possible? It\u0026rsquo;s because having one ADF instance you can switch between two modes: GIT integrated (for development purposes) and real instance.\nHowever, if you want to publish the changes (or new version) to another environment (or instance) - you must Publish the changes first. This performs to actions:\nPublishes the code from a developer version of code to real ADF instance. This can be done only from one branch: \u0026ldquo;collaboration\u0026rdquo; branch (\u0026ldquo;master\u0026rdquo; by default) Creates or updates ARM Template files into \u0026ldquo;adf_publish\u0026rdquo; branch. This branch will be used as a source for deployment. Deployment (#1 approach): Microsoft method (ARM Template) Then you can build your own CI/CD process for deployment of ADF, using Azure DevOps, for instance. I don\u0026rsquo;t want to dig deeper about how to deploy ADF with this approach as I already described it in the post: Deployment of Azure Data Factory with Azure DevOps.\nWhy many people (including me) do not like this approach?\nSemi-manual process, as at some point someone has to hit \u0026ldquo;Publish\u0026rdquo; button Full ADF (all artefacts) can be deployed only (no selective deployment) Limitation to one publish branch only (thankfully, you can name it now) Parametrize elements exposed within the ARM Template Parameter Restriction of 256 parameters maximum Building a release pipeline is not an easy thing Will not delete any existing ADF objects in the target instance, when the object has been deleted from the source ADF Must use a few tasks in Release Pipeline (Azure DevOps) to deploy ADF (including PowerShell script) I will discuss the above things more precisely below.\nI\u0026rsquo;m not saying that Microsoft\u0026rsquo;s approach is wrong. You may prefer that way. It may work for you.\nHowever, let\u0026rsquo;s have a look at how others are doing this:\n#2 Custom approach (JSON files, via REST API) There is another approach in opposite to ARM templates located in \u0026lsquo;ADF_Publish\u0026rsquo; branch. Many companies leverage that workaround and it works great. In this scenario, you don\u0026rsquo;t have to Publish the changes to update ARM Template. With this approach, we can fully automate CI/CD process as collaboration branch will be our source for deployment. This is the reason why the approach is also known as (direct) deployment from code (JSON files). In all branches, ADF is stored as multiple JSON files (one file per object), whereas in ADF_Publish branch - ADF is kept as ARM (Azure Resource Manager) Template file(s).\nDeployment (#2 approach): Directly from code method (JSON files) Why some people prefer this approach?\nIt\u0026rsquo;s much more natural and similar to managing the code of other applications Eliminates enforcement of using only one (adf_publish) branch (helpful if the company’s branches policy is much complex) You can parameterize any single property and artefact of the Data Factory Selectively deploy a subset of artefacts is possible Only one task in Release pipeline (Azure DevOps) covers all the needs of deploying ADF from code (more details below) What both have in common? In both cases, you must manage ADF triggers properly. Before deployment of any (active) trigger onto target ADF, it must be stopped, then deploy everything and start triggers again. This requires additional steps in a Release pipeline in order to do so. Microsoft offers PowerShell script to start/stop triggers as pre/post-deployment activity.\nMake your life easy Back in June 2020, I decided to write a brand new PowerShell module to help all of us when publishing the whole Azure Data Factory code from your (master) branch or directly from your local machine. It\u0026rsquo;s called azure.datafactory.tools, is an open-source and completely free module which you can download from PowerShell Gallery.\nDeploy Azure Data Factory extension to Azure DevOps in Microsoft Marketplace A few months later, on top of that, I created an extension to VSTS/Azure DevOps which allows you to do exactly the same things as a (one!) task in Release Pipeline. The task is called \u0026ldquo;Deploy Azure Data Factory by SQLPlayer\u0026rdquo; and can be found in Microsoft Marketplace.\nMore information on this page: azureplayer.net/adftools\nHash tag: #adftools\nThe rest of this post takes you through the aspects of using the approach of deployment ADF from code. Although you can write whole PowerShell code to deploy ADF on your own, I deeply recommend the usage either azure.datafactory.tools module or task mentioned above. It doesn\u0026rsquo;t make sense to reinvent the wheel, does it? Both tools will be called #adftools in the post below.\nDeploying objects in an appropriate order This problem doesn\u0026rsquo;t exist in Microsoft approach while deploying from ARM Template, you just don\u0026rsquo;t have to worried about it. However, deploying ADF objects (artefacts) one by one, you must care for dependencies between all of them. This is one of the most difficult things when you want to deploy ADF from code using RestAPI and your own script. #adftools does it for you automatically and it does it very well.\nPublish options There is only one method (or task) to publish ADF from code in #adftools. However, you have full control over the behaviour of the publication process. Publication parameters are used for this, which allow achieving all the below-mentioned goals. That\u0026rsquo;s the biggest and most meaningful advantage of the #adftools.\nStop/start triggers This issue is relevant for both approaches as always you must stop a trigger in order to update it. ARM deployment will not help you here. Therefore, you must take care of this before starting the deployment. In #adftools, there is a parameter (publish option) StopStartTriggers. It allows you to decide. It stops ALL existing and `Started` (Enabled) triggers. Generally, you should use TRUE (default) if you want to avoid troubles.\nHowever, if you choose FALSE - you must accept that process WILL NOT touch triggers. You can still do it for yourself in pre or post-deployment script.\nSelective deployment There are many different ways of how people are working with ADF in terms of its deployment. Some want to deploy only selected objects. That option is not available with Microsoft approach via ARM Template, it\u0026rsquo;s unsupported feature:\nBy design, Data Factory doesn\u0026rsquo;t allow cherry-picking of commits or selective publishing of resources. Publishes will include all changes made in the data factory.\nData factory entities depend on each other. For example, triggers depend on pipelines, and pipelines depend on datasets and other pipelines. Selective publishing of a subset of resources could lead to unexpected behaviours and errors. On rare occasions when you need selective publishing, consider using a hotfix. For more information, see Hotfix production environment.\nAlthough I agree that the concept is not easy, IMHO ADF Team exaggerates. As one of the options comes with #adftools is selective deployment. You can define a list of ADF artefacts you want to deploy by specifying them precisely by name (Includes collection), or (as an opposite), specifying which objects you do NOT want to deploy (Excludes collection). This is a very useful feature but would be a bit useless when you have to add each object to the list every time when new is created.\nTherefore, you can use the capability of defining objects by wildcards in its names or types. Furthermore, you can organise your objects in folders (in ADF) and selecting (or deselecting) them using name of folder(s) only. That will include or exclude all objects belong to the folder(s).\nThe syntax for pointing objects is: type.name@folder, where folder is optional.\nLook at the following examples:\ntrigger.* dataset.DS_* *.PL_*@test* linkedService.???KeyVault* pipeline.ScdType[123] trigger.*@testFolder Parametrisation / Custom parameters ARM Templates can be parametrized. Files created by ADF when you hit Publish button in UI have got parameters as well. The more objects you have in your ADF the more parameters will land in ARM Template. It is defined in the parameterization template of ADF which properties of which objects are being exposed. The parameterization template is in JSON format and hasn\u0026rsquo;t user-friendly syntax.\nToday, Microsoft shares example of that file for multiple objects, however working with the file is not one of the most pleasant things.\nSo, you may say: let\u0026rsquo;s leave default content and do not bother to modify the file? Sure. Yes, you can. In most cases, it would work.\nUnfortunately, sometimes you may be forced to do it:\nYou have too many objects in ADF and amount of parameters exceeds 256 Some of the properties you want to change during the CI/CD process are not parameterized by default In such cases, use the custom parameter file and remove properties that don\u0026rsquo;t need parameterization or extent it by adding the right section(s) for selected objects and properties. Do follow the instructions on Microsoft documentation to discover how \u0026ldquo;easy\u0026rdquo; it is.\nSometimes, instead of adding parameters to the template, you can slightly change ADF by adding global parameter(s) and leverage it wherever needed. That minimises the efforts, still keeping the solution clear and understandable.\nUsing #adftools, you will not even bother with parameterization template. Because that \u0026ldquo;other\u0026rdquo; approach uses JSON files as a source of deployment. In that scenario, we don\u0026rsquo;t know and don\u0026rsquo;t care what\u0026rsquo;s in \u0026lsquo;adf_publish\u0026rsquo; branch, because we do not use it. Our source is main, master or another collaboration branch, which contains the version of ADF ready to deploy.\nHow to override parameters then?\nIt\u0026rsquo;s very easy. #adftools has a build-in mechanism to replace the properties with the indicated values (CSV file). You must create a configuration file (CSV or JSON format) and define all properties of objects which value need to be replaced (optionally they can be added or removed). The good thing is you can define an absolute minimum set of objects. Another good thing is: you can use wildcards to specify multiple objects if they have recurring value pattern.\nAn example of the file may looks as follow:\ntype,name,path,value # As usual - this line only update value for connectionString: linkedService,BlobSampleData,typeProperties.connectionString,\u0026#34;DefaultEndpointsProtocol=https;AccountName=sqlplayer2019;EndpointSuffix=core.windows.net;\u0026#34; # MINUS means the desired action is to REMOVE encryptedCredential: linkedService,BlobSampleData,-typeProperties.encryptedCredential, # PLUS means the desired action is to ADD new property with associated value: linkedService,BlobSampleData,+typeProperties.accountKey,\u0026#34;$($Env:VARIABLE)\u0026#34; factory,BigFactorySample2,\u0026#34;$.properties.globalParameters.\u0026#39;Env-Code\u0026#39;.value\u0026#34;,\u0026#34;PROD\u0026#34; # Multiple following configurations for many files: dataset,DS_SQL_*,properties.xyz,ABC You can decide to keep one config file per environment or create only one file and govern all values in Azure DevOps Variables. Both ways are OK, just remember to NOT keep sensitive information (credentials, keys, secrets, passwords, connection strings which contain password, etc) in plain text in the config file. For such properties, put placeholders like $Env:VariableName and keep values securely in Azure DevOps Variables or read them from Azure Key Vault prior.\nMore information about config files in the documentation of azure.datafactory.tools PowerShell module.\nDropping objects not existing in the source any longer Publish options in a Release pipeline task This option simply does not exist in Microsoft approach. Publishes will include all changes made in the data factory, albeit it will NOT delete any objects (dataset, pipeline, etc.) removed from ADF by a developer in Git mode.\nTherefore, when you delete any object in ADF - it will disappear from code repo, but still is present in ADF service, even after deployment. You must delete these objects manually. If you leave such a mess for some time, you may find yourself in a situation where many objects need to be removed. Deleting objects in ADF requires taking all dependencies between objects into consideration to be able to delete them in the correct order. This may make this task time-consuming and not easy.\nNot to mention that the publishing process should be fully automated, without manual interference and rummaging through ADF objects (especially in production).\nHence, it\u0026rsquo;s another advantage of using #adftools rather than deployment ADF from adf_publish branch and ARM Template.\nObviously, it\u0026rsquo;s only an option and by default is disabled.\nIn the PowerShell module, there are two options in PublishOption class:\n- DeleteNotInSource - Indicates whether the deployment process should removing objects not existing in the source (code)\n- DoNotDeleteExcludedObjects - Specifies whether excluded objects can be removed. Applies when DeleteNotInSource is set to True only.\nBoth have their counterparts in Task extension to Azure DevOps (see picture on the right).\nAs I mentioned above, #adftools offers filtering objects intended for deployment by using \u0026lsquo;Includes\u0026rsquo; or \u0026lsquo;Excludes\u0026rsquo; collection, or Filtering options in DevOps Task.\nIn that case, is worth to remember an excluded object can be deleted only if flag `DoNotDeleteExcludedObjects` = false.\nThis situation could cause an error when a trigger has not been disabled prior and is being deleted.\nWhen flag `DoNotDeleteExcludedObjects` = true, nothing will happen to exclude objects.\nThis option gives you a flexibility of deleting objects in the target, but still not touching objects from \u0026lsquo;other\u0026rsquo; group.\nRecap Let\u0026rsquo;s summarize what we\u0026rsquo;ve learnt in this post. All this information can be very confusing, therefore quick overview might be helpful here. Let me compare both deployment methods in the table for better understanding.\nThe method recommended by Microsoft Deployment from code Deployment method ARM Template RestAPI / PowerShell Source location ADF_publish branch. Only one per ADF Master or any other branch. Multiple branches possible Source files ARM Template files Artefact JSON files Custom parameter syntax arm-template-parameters-definition.json Not required Input parameters ARMTemplateParametersForFactory.json Config file or files CSV or JSON format Number of parameters Limited to 256 No limit Selective deployment Unsupported Available by filtering Dropping objects not existing in the source Unsupported Available as option Visible in Deployments (Resource Group) Yes No I hope this post helped you to understand what kind of alternatives you have when it comes to publishing Azure Data Factory and you understand the differences. It\u0026rsquo;s up to you which method you\u0026rsquo;ll choose, probably based on your preferences and skills available in your team.\nGood luck!\nUseful links Microsoft documentation: Continuous integration and delivery in Azure Data Factory\n#adftools - Depends on your preferences, you can use:\nazure.datafactory.tools PowerShell module: GitHub code \u0026amp; doc | PowerShell Gallery\nor extension to VSTS/Azure DevOps: Deploy Azure Data Factory by SQLPlayer.\nWho mentioned the #adftools Azure Data Factory CI/CD with GitHub and Azure Pipelines\nCI/CD for Azure Data Factory: Create a YAML deployment pipeline\nUsing Azure DevOps CI CD to Deploy Azure Data Factory Environments\nContinuous Integration and Continuous Delivery in Azure Data Factory V2 using Powershell: Part 1\n","date":"2021-01-15T20:25:08Z","image":"/uploads/2021/01/publish-adf-custom-ci-cd-approach.png","permalink":"/2021/01/two-methods-of-deployment-azure-data-factory/","title":"Two methods of deployment Azure Data Factory"},{"content":" Introduction Alex is a Data Platform MVP who loves DevOps.\nHe\u0026rsquo;s has been helping data professionals apply DevOps principles to relational database development and deployment since 2010. He\u0026rsquo;s most proud of helping Skyscanner develop the ability to deploy 95 times a day. Alex has worked with clients on every continent except Antarctica - so he\u0026rsquo;s keen to meet anyone who researches penguins.\nA keen community member, he helps organise Data Relay and he created speakingmentors.com. He blogs at workingwithdevs.com, speaks wherever they\u0026rsquo;ll let him and manages the DLM Digest quarterly email: a report on the latest database DevOps news/tutorials. He was awarded his first MVP award in 2017.\nDue to the coronavirus outbreak, this talk has taken place online on 10 July 2020 (Friday) using Zencastr platform.\nInterviewer: Kamil Nowinski (T) \u0026amp; Michal Sadowski (T).\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript KN: And how do you prepare yourself for a speech, for talking and leading the technical session?\nAY: How do I prepare for doing a tech session? I drink a lot of apple juice, which sounds silly.\nKN: Does that help?\nAY: Yeah. It’s probably rubbish pseudoscience but somebody told me at a course that apple juice has a fairly unique quality. They said that the pH level is exactly like what it should be in your throat. I’m not sure I believe that but I’ve never looked into it in that much detail and I’ve never wanted to because it works and I don’t want to pull the empress clothes off it. It seems to work for me. Apple juice has this unique effect that it really, really does soothe your throat compared to water, which just kind of dries it out. So yeah, I drink a lot of apple juice. And if I’m presenting from home, especially if I’m doing a long presentation, like a pre-con or like a full-day training session, and that really, really does hammer my voice. Often, after I’ve done a tour or a three-day training class, I do end up losing my voice at the end of it. So on those days, I will have like an entire carton of apple juice per day as I’m delivering the session, if it’s like a full day thing. So apple juice, absolutely.\nKN: So that’s from your voice perspective.\nAY: If I’m nervous, I think of the butterflies flying in formation, which is an old acting trick that I got taught back when I was an actor in theatre and stuff. Those butterflies are there to help you in your fight and flight. They’re not there to distract you. They are there for an evolutionary purpose to help you perform better, not worse. All you need to do is learn how to handle them. If you get butterflies and it makes you feel skittish and it makes you feel sick, think about them, control them, get them to fly in formation and that gives you a fire in your belly, and it becomes a force you can use. So yeah, that’s one of my favourite tips. But in terms of the more practical stuff, I will often spend a couple of hours rehearsing in my room beforehand, and I will have done dry runs. I try to always record myself at least once before I deliver it live. Everybody hates watching themselves back. I hate watching myself back. I hate the sound of my own voice, just like everybody else does. But I think it’s a really useful thing to do because you are your own worst critic, and so when you play it back, it helps you to see “oh, that’s annoying when I do that, I should try and do that a bit more quickly, I’m waffling here”. I do tend to waffle, I’ve probably waffled a lot in this podcast and I’m sorry for that, but when you’ve got a kind of scripted thing, it allows you to focus on “right, OK, I really need to get to the point quicker here”. Or actually, there’s a key piece of information that I missed here and it made it hard to follow the narrative. So watching yourself back is important. But also from plain presentation style type things it is good. You can see that I’m always looking down or I’ve got this weird habit that I do with my hands and it’s off-putting, or I tend to mumble or whatever it is. You just see kind of plain presentation stage presence-y type things will pop out at you a lot more. The caveat I’d say to that one is, if you’re the sort of person that is very, very critical of yourself and if that’s likely to do more harm to your nerves than it is to give you the confidence, then don’t do it, because nerves are a real thing and they’re important. But if, like me, that helps you to spur you on and helps you do better, then I absolutely recommend watching yourself back. What else do I do? I’ve started always using the PowerPoint subtitles now. The audio caption stuff. People always ask me “how do you do that?”. It’s literally a button in PowerPoint. You go to\u0026hellip; I can’t remember the name of the tab in PowerPoint anymore but it’s like the presentation tab. There’s like a check box that you click, and it will start giving you subtitles for everything that you’re saying. There’s an option “Do you want to appear at the top or the bottom?”. You put the top, because if you’re in a big crowded room, then people might not be able to see the bottom of your screen but everybody can see the top. And on your PowerPoint template, what I do is I put a big orange box right where the subtitles are going to be, and I just put the title of the presentation in that box on every single slide. And what that means is that I don’t end up having the subtitles covering up important information on my slides, because whenever I create a new slide it’s like “right, I can’t put anything in this box because that’s where the subtitles are going to be”. So it forces me to design my slides with my subtitles in mind.\nKN: Interesting.\nMS: You’ve just described all of the technical skills. So for the technical skills you have books, conferences, but what about the soft skills. You mentioned at the very beginning about mentoring, so how do you learn that?\nAY: Soft skills. I mentioned a few soft skill tips. I probably mentioned more soft skill tips than tech skill tips so far. I think the best thing you can possibly do if you do struggle with nerves is to find somebody to be an ally or a coach or a mentor. Somebody to speak to about it, somebody to encourage you, somebody who you know has got your back, somebody who can give you feedback, somebody who you can receive feedback from in a healthy way, rather than getting frustrated or angry about it. You can actually say “right, OK, well, thank you for the feedback, I’ll try and do something about that”. Somebody that’s gonna make you feel good about yourself. The best thing you can do is find that ally or mentor or coach who can help you to be the best version of yourself and to have the confidence. Because it is a nerve-wracking thing. That’s one of the reasons I set up speaking mentors, as a way of helping people to find the person who can give them the support they need. And what I actually find is that for most people it’s that confidence boost. So my best piece of advice is find somebody, probably somebody local who goes to a lot of the same conferences and user groups that you do, because nothing beats that.\nKN: And what about your work-life balance? Having a small baby boy and working as a consultant, that must be very difficult.\nAY: So when Emma got pregnant, I deliberately changed the business model of DLM Consultants to give me a much more reliable income. Because beforehand, Emma had the reliable income that we knew came in the same every month, and my income was\u0026hellip; Emma paid for the important things and I paid for the nice things. And obviously, when we had the baby, and Emma takes a year out of work, I need to pay for the important things, which means that the risk profile changed somewhat. And also it means that I really don’t want to be going away from home as much. It now has been two years since I took a flight for a piece of paid work. That’s crazy. If you told me two years ago that I wouldn’t take another flight for a piece of paid work in the next two years, I would not have believed you for a second. My first two years at DLM consultants I barely earned a penny without getting on a plane. So yeah, to have primarily remote contracts, to focus more on longer-term mentoring contracts as opposed to short-term “let me help you build a pipeline” type things. When I started doing those, my work got a lot more predictable and I was able to do it from home. And as it happens, as time would tell, it also set me up much better to be defensible from COVID. So as COVID has happened, I’ve been able to more or less carry on work as normal. The main difference has been the lack of child care, meaning I need to reduce my hours. But I’m able to do that because I’m able to work from home and pick my own hours, which means that I’ve got much more flexibility about how I handle my work/life responsibilities.\nKN: So it’s particularly hard to start your own business, especially at the beginning, first one or two years.\nAY: Absolutely. We were in a very fortunate position that when I started the company, we didn’t have any children and Emma had a good and reliable job, which meant that based on the savings we had, we could survive without me earning or with me earning a nominal amount, without making any significant changes to our lifestyle. Basically, it meant that we couldn’t put the same amount of money that we were doing into savings and we can kind of save houses and holidays and all those sorts of things. We had to stop that but our day-to-day lifestyle we were able to not change that too much. And we would have been able to fight like that for quite a long time, because we were in a very fortunate position. Had that not been the case, I definitely would have struggled to start. It would have been a much, much scarier thing to take that leap had we not been in that position.\nMS: What hints would you give to young people who wanted to start working on the market. So they just graduated and they would like to start on the IT market.\nAY: Make things, do something. Even if it’s not a work thing, get involved in a pet project, get involved in some open-source project, build stuff for fun, build a chess programme, build a poker programme, just make stuff and learn about how to make stuff and what goes wrong when it does. Because you’re probably going to be interviewing for junior developer roles or maybe that’s not the type of career you want but you’re going to be applying for some sort of junior IT role. So in that interview, you want to be able to talk about the things that you’ve made, to make stuff. One of the things that I made while I was working for Redgate, while I was learning stuff, my brother is a software developer and we worked together to try and create\u0026hellip; It was an algorithm to play tic-tac-toe, kind of noughts and crosses. But the way that we created it is by kind of evolving it by simulating evolution to try and get it to grow itself. So we basically came up with a random generator that would randomly create the logic, various different logic\u0026hellip; We basically came up with a way to encode the logic as a random, long binary string, and then we just generated random binary strings, we made them play against each other, and we left it running for a while until eventually it came out with a tic-tac-toe bot that was quite good at playing tic-tac-toe. So that was one of the things that I did as a fun little side project when I was working for Redgate, and it was a lot of fun. So make stuff. And if you like it, wonderful. If you enjoy it, wonderful. You’ve probably found the right sort of career. If you really don’t enjoy that, then maybe a career in IT isn’t for you. If you don’t enjoy picking the projects that you want to build, then maybe you’re not going to enjoy it when somebody else asks you to build something that you’re not particularly passionate about. So yeah, just make stuff, make stuff, make stuff. Make lots of stuff, and if you enjoy it, great, and it’ll give you something to talk about in your interviews, and if you don’t enjoy it, maybe try something else.\nKN: That’s a very good hint. And you are co-organizer of Data Relay so I hope all folks from the UK are aware of that free event. But please, could you explain the concept of that for all who isn’t.\nAY: So Data Relay is not going to be running during COVID because it’s the worst possible thing you could do during COVID but once we’ve got past COVID, and for the last 10 years before we had COVID, what we would do is we would put on a local free Data Platform training event, and we would run it in multiple cities on consecutive days. So last year for example we ran an event on Monday in Newcastle, then on Tuesday we went to Leeds, on Wednesday we went to Nottingham, then Birmingham, then Bristol at the end of the week. In the past we’ve done slightly longer and slightly shorter versions, but we’ve now settled on five events in five days as being a fairly good sort of size and scale for the event. They’re free events, they typically have three tracks of regular sessions and one track of half day workshops for more in-depth, technical, pre-con style classes. At least one of those at each location is normally run by the folks from Microsoft, so you’re learning from Microsofties about how to do cool and exciting stuff with data in Azure. So that’s a lot of fun. And the whole concept is to bring free and accessible training to you. So that’s why we go to so many different cities. And we try and cover as much of England as we can, or England and Wales. We don’t tend to go up to Scotland because they’ve got some great events of their own and they’re doing good. We’ve not tended to go over to Northern Ireland or the Republic, because there’s kind of a sea in the way so it’s a bit hard. So typically we do England and sometimes we’ll go to Cardiff. But yeah, it’s a lot of fun. And for those people that want to join us for the road trip, we put on a bus for speakers and organizers and helpers to kind of go from city to city, which is exactly why it’s the worst possible thing to do during a pandemic, because I can’t imagine anything worse than having a 200-person event in one city and then another city, and another city, with a bunch of tired and probably immuno-compromised people all sitting together on a bus between each city. It sounds like the worst possible thing to do during a pandemic. So we won’t be doing it again until COVID is fixed.\nKN: Which basically we don’t know when it will happen.\nAY: Yeah, that’s why we don’t currently have any plans or any dates in mind, and we won’t have until such a time as it looks like it’s under control.\nKN: And due to this pandemic, this year Data Relay has been cancelled because SQLBits has been moved to that period of time, right?\nAY: To be honest, I think possibly when that happened Relay dodged a bullet. The original SQLBits was going to happen in February or March, which was right in the height of the pandemic and when everything was being shut down. So they clearly couldn’t run at that time. (SQL)Bits were offered three dates: one in the middle of the summer holidays, which wouldn’t have worked for various reasons, one basically over Christmas, which wouldn’t have worked, and a third one, which was smack in the middle of Data Relay. So that’s why they rescheduled to October. And they told us in advance of the announcement and they asked us\u0026hellip; They were really, really good, they arranged a call with us and we discussed it. And we agreed that we don’t want to stop SQLBits from happening, we don’t want to get in the way of SQLBits, we don’t want to compete against SQLBits. We actually want to attend SQLBits. And SQLBits were in a really difficult financial situation. They basically had the option to cancel the event and bankrupt themselves or they had three dates that they could go for. So they went for that one. And all of this, by the way, was agreed right at the very beginning of the outbreak, before any of us knew anything like as much as we do now. So at that stage we’re like “yep, fine”. We hadn’t got very far of our planning, we paid one or two deposits for one or two of our venues but we possibly only spent five or ten percent of our budget for the whole thing. So we were early enough that we could call it off without taking a massive financial hit, whereas SQLBits was kind of walled in on it. And SQLBits has a much, much bigger budget than Data Relay does. So I think it would have been enormously mean of Data Relay not to say “look, this year is just things happened”.\nKN: Just a special situation. And I think the solution was pretty good, was very good.\nAY: As I was going to say, we probably dodged a bullet because had that not happened, I suspect Data Relay probably would have ended up postponing or cancelling anyway, but we probably wouldn’t have done it for until a couple of months later, by which point we would probably have spent more of our budget. So in a sense it probably saved us from ourselves. I think at the end of the day, Data Relay 2020 as it was originally planned would have been cancelled sooner or later anyway. We didn’t realize that at the time but we do now. As for when it does run again, I don’t know. Originally, we were thinking about “do we run a November thing, do we run a spring thing, do we wait until October 2021?”. All of those things are up in the air but based on what’s happened, we’re just not even gonna start thinking about or making any plans until we all know with confidence that the thing is finished, that COVID is finished.\nKN: What do you think about SQLBits? Will it happen or not? Because we are closer and closer to the date and the pandemic is still with us.\nAY: Don’t ask me that question. I don’t know. What are the SQLBits committee saying at the moment? Because I believe that whatever the SQLBits committee is saying is exactly what will happen. I’m not going to try and second-guess their choices. I can imagine that it cannot be easy to be in their shoes. Will it be safe to run SQLBits in October? Will Americans be able to fly over without self-isolating?\nKN: Tons of questions.\nAY: Yeah. Given that Microsoft is one of the biggest investors in SQLBits and a lot of the speakers are going to be American, I suspect that the ease and safety of travel between the USA and the UK is probably going to play a significant factor in SQLBits, I would imagine. If it’s not possible or practical, or safe for Americans to take part in SQLBits this year, what does SQLBits do about that? I don’t know. Do they find another 30% of the speakers again to replace all the Americans?\nKN: Probably not, or it would be very hard. Let’s leave this question unanswered.\nAY: If you go back six months a lot of people were talking about “are you going to go to PASS Summit or SQLBits?” because a lot of people were doing the maths and figuring out that actually even for an American it’s cheaper to go to SQLBits than it is to PASS Summit. So I don’t know how many attendees were going to be expected from the US as well. So yeah, I don’t know, and I feel for the folks that organize it and I wouldn’t want to make any judgment about what decision they would ultimately choose to make.\nKN: OK, let’s move on to another question.\nMS: From the technology perspective, what is the biggest challenge in IT these days?\nAY: What’s the biggest technical challenge in IT? It’s not the question you asked but I’m gonna come back to\u0026hellip; No matter what you ask, it’s always a human problem. I actually think that the biggest technical problem in IT is the humans who are writing it. And if you can solve those human problems, then I suspect that you’ll do a lot better. I think you’ll get a lot more bang from your buck from solving a lot of the human and organizational problems. And I think you’ll get more bang for your buck for doing that than you will for most technical problems. However, to answer the question you asked me rather than the one I wanted to answer, what is the biggest technical problem, I think security and data protection within the DevOps world I think is something that people have been gradually waking up to. I mean, everybody thought that GDPR was going to be the thing that was going to do that and it’s not really done it to the level that people thought it was going to, but still, I think that as data grows and people want to be able to test that their changes work on larger quantities of data, the provisioning of development and testing databases that contain appropriate test data that does not contain sensitive production data but which does look and feel like production data and have some similar scale to it. I think that those problems are going to be really big. Related to that, I still see a lot of people using shared development servers for database development, which I think is the root cause of so many of the bad practices that you see in database source control and deployment.\nKN: Also, the cause of many mistakes and potential problems with deployment.\nAY: Absolutely, it’s the root cause for so many different reasons. It’s tangled into branching and self-service development environments and it works on my machine, all of those things are related.\nKN: Let’s go back a little bit to the topic about DevOps and databases then. The topic always discussed and maybe not very well understood by people sometimes is “state versus migration”, SSDT versus Redgate or other tools. Which approach is better? I remember that probably one or two years ago you compared those approaches, so which approach is better?\nAY: It depends, it absolutely depends. The reason I started writing those blog posts about it\u0026hellip; To explain the core debate that we’re talking about right now: when you’re source-controlling a database, you can either use something like SSDT or Redgate SQL Source Control that is going to define how the database should look. And then you’ll use some sort of schema comparison tool like SQLPackage or Redgate SQL Compare or something like that, to generate an upgrade script and deploy it. And then, when you deploy it, you always know the production matches your source code. The problem with that is the comparison engine that you’re trusting, and whichever one you’re using, whether you’re using the Microsoft one or the Redgate one or something else, they’re never perfect. So you need to design your deployment pipeline with the acknowledgement that something could go wrong at some point, so you need to be testing it, you need to be validating it, you need to be running it in dry run environments, and you need to be alerting if anything ever goes wrong, and you need to have some sort of plan B. In many ways, I think of state- or model-based deployment as the 80/20 rule. It’s actually easier to set up and manage a lot of the time, and it’s a simpler developer experience a lot of the time, and it means that you don’t need to worry about writing the upgrade scripts yourself a lot of the time, and 80% of the time it makes your life an awful lot easier. But 20% of the time you end up having to solve the difficult problems. So it’s one of those. You can get 80% of the benefit with 20% of the effort if you go for a state- or model-based approach. However, if what you want is something that’s going to work all of the time and if you want something that’s really reliable, you’re probably going to go for more of a migrations approach. Now, migrations approach is basically a series of update scripts. Script 1, script 2, script 3, script 4. You run them in order. Typically, it’s combined with some sort of deployment history or schema migration log table on the target database, which records which scripts have been run, so when you run your deployment, you just run anything that’s not yet been run, in order. And that sort of approach frees you from relying on the comparison engine. It means that if you’ve had experience with Redgate or SSDT creating upgrade scripts that either don’t perform very well or they don’t solve various technical problems, like deploying a new NOT NULL column without a default or renaming something and it didn’t realize it’s a rename, it thinks it’s a DROP/CREATE and you lose data. It solves you from these sorts of problems. But at the expense that you have to manage these migration scripts. At small scale that is not a big problem. So at small scale migrations tend to make a lot of sense. If what you have is a dumb data store which doesn’t have very many changes going to it very frequently, a small migration script runner with a few dozen scripts is a relatively neat and reliable solution to that problem and it’s not very difficult to understand. But once you start scaling up your development and you’ve got large teams of developers always making changes and maybe you’ve got one of those horrible monolithic architectures where you’ve got one big SQL database in the centre of everything that backends 150 applications and you’ve got different people from different teams all over the place trying to update it with different source control and release cadences. At that point a migration-based approach becomes really, really difficult to manage, and it just becomes this huge repository full of all sorts of scripts and then kind of ordering and the scheduling, and the coordination of these scripts, there are a lot of problems associated with that. To summarize, the model-based approach will get you most of the way there for less effort, and it can scale better sometimes but it’s never going to be a complete solution. There are all sorts of fixes to that which are never particularly nice, like using pre or post deployment scripts or something like that to solve the nasty issues when stuff goes wrong. On the other side, you can go to a full migrations-based approach, which is probably more reliable, but once you start hitting any sort of scale, it becomes a big administrative overhead problem. So yeah, which is the right one? It depends.\nKN: Yeah, exactly, to be honest with the people, both approaches have their pros and cons. There’s no perfect solution.\nAY: They do. And you know what? What you just said about the people, a lot of the time look at the people in your team. If your team all know and understand how to use SSDT and have got experience with it and know how to handle the issues when it goes wrong, then why would you pick something else? Because they already know how it works. Stop worrying about your source control process. Let’s set it up and focus on your product. And the same goes for if they’re used to using a migrations-based approach. If your team are all used to using migrations and they’re comfortable with it, just the fact of what the team are comfortable using is probably one of the most important factors.\nKN: That’s absolutely true if you’re already using something or if you have some specific skills in your team, that’s absolutely right. But I’m just wondering if someone is just starting that journey and setting up that solution, that’s when that question comes up, which solution to choose.\nMS: How does cloud fit here? What do you think about the cloud-first, cloud-only approach? So all of the approach with the database, with the migration or the state approach and then also the cloud. Is it a perfect match or is something different?\nAY: Yes, I’m a big fan of the cloud. I think that a lot of people think the cloud is going to solve all of their problems. That’s not true, it’s not going to solve all of your problems, there are going to be new problems going to the cloud. But fundamentally, I think that the cloud is probably a better place to host your stuff. Whether that means you should be lifting and shifting and putting everything there immediately or not is a different question. Whether or not you should necessarily be migrating your mainframe application to the cloud, probably not. I think that the cloud definitely is going to give us huge advantages and already is doing, but it’s going to disproportionately reward applications that are designed cloud-first from the beginning rather than migrating stuff that is already running on-prem. I think that there is a tendency to want to lift and shift stuff into the cloud, and if your application was not designed with the cloud in mind, then you’re not necessarily really going to reap the benefits. The cloud is definitely a good thing and definitely stuff is going to be going there. Do I think that it’s the right thing and we should all be immediately going there right now with everything? No. I think that there will be a lot of people who will choose not to go to the cloud, and I think there’ll be a lot of applications that are running today that never will go to the cloud. And I think the decision has to be made on a case-by-case basis.\nMS: What is the hottest feature of SQL Server or the technology that you are learning at the moment, the first item on your learning list?\nAY: So at the moment, I’m doing quite a lot of learning some Infrastructure as Code type stuff on AWS, just because I need it for a project that I’m working on at the moment for one of my customers. So I’m playing a lot with Octopus Runbooks and AWS PowerShell and CloudFormation and stuff like that to spin up some infra. It’s not particularly new tech, it’s just tech that I’ve not personally played with that much because I’ve spent a lot more time at the database level, so now I’m not an expert at all by any means in it. I’m just beginning to pick it up and figure out how I build out my infrastructure in AWS, because I need to for a project. And I’m enjoying it because it’s probably long overdue. So that’s what I’m personally working on at the moment. What am I excited about learning, I am very excited to see how Kubernetes and SQL Server are going to play out. How we’re going to start hosting SQL Server in future years, so I’m really, really interested to see what happens there. So what do I want to be learning right now? I’d probably love to be doing a lot more Kubernetes than I’m doing right now. But what am I actually doing? I’m doing a lot of basic automation of EC2 spinning out VMs and stuff, just because it’s long overdue and it’s not as exciting as it might have been five or ten years ago. But it’s what I need to do right now for the project I want at the moment.\nKN: Cool! OK, so almost at the end of our conversation, I need to ask you about that because Michał and I, we are Polish guys, so I need to ask you, because I remember that you have some Polish roots, don’t you?\nAY: I do have some Polish roots but they’re several generations past. So my grandparents ended up in London at the end of the Second World War as refugees. So my mother was born in 1948 I think in London to a pair of Polish refugees who were a young married couple who had just been through the Second World War together. So I do have a large branch of my Polish family, because they all seem to have about 15 children, so I have a very, very, very large Polish family that seems to cover a broad spectrum of Polish society and we were actually scheduled to have our five-yearly zjazd (reunion) in Poland this year. It got cancelled because of COVID but hopefully we’re going to do it next year instead. And each time we go, it feels like there’s 30 or 40 new little children. I think at the last one, there were about 60 people on the zjazd, and based on expectations, I think there might be over 100 this time, of which the vast majority of them are Polish.\nKN: So, I’m guessing that you’ve been to Poland several times at least?\nAY: Oh, yeah. I’ve been to Poland a few times. Unfortunately, I don’t speak the language and I feel very guilty about it but as I say, I’m myself second or third generation, I’m not sure how you count it. My dad’s English and you would never know my mom was Polish.\nKN: Don’t worry, we can teach you some words. For example, at the end of this conversation, you can say in Polish, like “goodbye”, you can say “do widzenia”.\nAY: Do widzenia, dziękuję.\nKN: OK, Alex, at the end of our conversation, could you tell us where we can find you, where the people can find you in the Internet, how can they follow you?\nAY: The main social media I use is Twitter. I’m _AlexYates_ on Twitter. I don’t really use Facebook anymore. I’m very close to deleting my Facebook account. I actually would have done, had there not been a particular thing for my kid that we do on Facebook, which I’m not going to go into. You can get me on LinkedIn, but normally I’m not very good at responding to messages there because it’s mostly recruiter spam, so the best place to get me is Twitter. Or go to the website dlmconsultants.com and if you want to contact me about potential consulting or mentoring relationship, then dlmconsultants.com and there’s an email address you can use there to contact the company.\nKN: Cool, fantastic. Thank you very much again, Alex, for the conversation today and for your time.\nAY: Thank you very, very much for having me. Dziękuję!\nMS: Thank you.\nKN: Dziękuję, do widzenia!\nUseful links Alex\u0026rsquo;s profiles \u0026amp; websites: Twitter | DLM Consultants | Speaking Mentors | Blog\nConferences: Data Relay | DataGrillen | GroupBy | SQLBits\n","date":"2020-12-18T08:00:16Z","image":"/uploads/2020/10/Alex-Yates-SQLGrillen-MVP-award-2017.jpg","permalink":"/2020/12/asf-035-alex-yates-interview-part-2/","title":"ASF 035: Alex Yates interview (part 2)"},{"content":" Introduction Alex is a Data Platform MVP who loves DevOps.\nHe\u0026rsquo;s has been helping data professionals apply DevOps principles to relational database development and deployment since 2010. He\u0026rsquo;s most proud of helping Skyscanner develop the ability to deploy 95 times a day. Alex has worked with clients on every continent except Antarctica - so he\u0026rsquo;s keen to meet anyone who researches penguins.\nA keen community member, he helps organise Data Relay and he created speakingmentors.com. He blogs at workingwithdevs.com, speaks wherever they\u0026rsquo;ll let him and manages the DLM Digest quarterly email: a report on the latest database DevOps news/tutorials. He was awarded his first MVP award in 2017.\nDue to the coronavirus outbreak, this talk has taken place online on 10 July 2020 (Friday) using Zencastr platform.\nInterviewer: Kamil Nowinski (T) \u0026amp; Michal Sadowski (T).\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript KN: Hello, Alex! How are you? Good morning!\nAY: I’m good, thank you, I’m good. How are you?\nKN: Very good, thanks! Thank you for accepting our invitation to this podcast Ask SQL Family. Could you introduce yourself at the beginning? What is your name and where do you live?\nAY: OK, cool. My name’s Alex. I live in Cambridge in the UK. Alex Yates. I’ve been helping people with DevOps and specifically to solve the problems that data poses to DevOps for the last 10 years. I started that journey at Redgate for a few years as a salesperson and pre-sales engineer. And in 2016 I quit and started my own company DLM Consultants. I also do a whole bunch of community stuff. I’m one of the organizers of Data Relay, which is a big 1000-person five-city conference in the UK. We run that once a year, except for this year, because COVID. And I set up a website called Speaking Mentors a while ago, because we need more speakers.\nKN: We will ask about all these things but before we go into that, I would like to ask you what you’re doing for a living specifically right now. I mean, I would like to have or hear more details because DevOps is a very wide topic.\nAY: So what’s the question? Specifically, what am I working on at the moment?\nKN: Yeah, for example, yeah. What kind of work are you doing under this DevOps phrase?\nAY: Cool, so the primary part of my work is helping my customers to basically make their database change management and deployments easier. DevOps is a big, broad concept that touches on lots and lots of things. And my primary technical expertise is to help people source control and deploy databases, because most people aren’t very good at that. So that’s what I do primarily. But more broadly than that, I also mentor and coach people with general DevOps principles, because as much as the technical stuff can be hard for database source control and stuff, actually it is just details and what’s much more important is people understand the underlying reasons why we’re source controlling databases and how that fits into the bigger DevOps picture. So I do a lot of coaching and mentoring at a slightly higher level, so that folks can understand the importance of continuous delivery pipelines but also the importance of kind of feedback and developing that DevOps culture within an organization.\nMS: Can we take one step back and could you please describe what are the three advantages of DevOps for someone that is not familiar with DevOps?\nAY: So the three advantages of DevOps\u0026hellip; There are quite a few.\nMS: There can be 5.\nAY: OK, so the high-level objectives is it helps you release your stuff more quickly, more reliably and more cheaply. That’s the headline feature. The point is that we’re trying to change that vicious cycle where deployments go badly, so people try and wrap more bureaucracy around them and do them less frequently because they’re scared of them. But then the problem is that they become even bigger and even more complicated and they’re all more likely to fail. So it’s this counterintuitive thing where in order to make your deployment safer, you need to do them more frequently, which is a weird jump for a lot of people. And there are a whole bunch of reasons that I could talk about all day why that’s important, but ultimately I help people to safely release more frequently and to reduce the overhead and any manual work that needs to be done to get a release out the door. Because the easier it is to deploy, if it can be a kind of “click a button, off it goes” or if it can even be fully automated, that frees you up to release more frequently with less overhead, which means your deployments become smaller and safer, and more reliable. And also it means you don’t have to invest six months building a thing before you put it in front of your customers. If you can change the way that you work into much, much smaller features, then you can start delivering on your ideas much, much more quickly, in a single sprint or even just a few hours. The perfect ideal is you have an idea, you have a hypothesis, you want to test it out, you release one version to some people, one version to other people, if it’s a really, really small thing, you could put the code together in a couple of hours, get some feedback on it in near real-time and then iterate. That’s the ideal. Obviously, a lot of people are a long way away from that, and in database land that almost feels impossible to a lot of people. That’s because you’re standing at the bottom of Mount Everest and staring at the top of it. So I help people to start taking those first few steps to start heading in the right direction.\nKN: So what do you think, what is the maturity level of DevOps for the organization on average?\nAY: On average?\nKN: I know that Redgate has an annual report about that state and this information is divided by sections.\nAY: I’m always nervous when we start worrying about maturity models because it kind of gives this prescriptive idea about what DevOps looks like. And ultimately, I feel to a certain extent that’s putting the cart before the horse. Like, the objective is to do whatever your business needs to do. It’s to do whatever wonderful idea you have and to get it out there quickly and reliably, and safely, and to get feedback on whether it’s working so you can iterate and have your next idea. That’s what we want to do. So if we define it as a maturity model, what that often ends up looking at is “are you source controlling this way, are you deploying this way?”. You need to do the things in a prescribed order, which I’m not sure that’s that valuable. It’s like you’re doing the things because the book tells you to do the things or because you have to do source control first and then you do this first, and then you do this. And it seems to me to both be more beneficial and also easier to sell if you focus on things the other way around. So what are your business objectives and let’s make sure we meet those.\nKN: And that perspective also I think is much easier to be understood by the business.\nAY: Exactly. How many times do people have trouble in IT because you have technical geeks and I’m going to say “the business folks” in inverted commas, because I don’t like describing the business as “other” than IT because IT is part of the business. But fine, you tend to have the commercial senior exec folks who don’t really understand IT and can’t speak tech language, and you have the tech folks who are very, very good at understanding the inner workings of SQL Server but don’t understand the business.\nKN: That’s what I meant, yeah.\nAY: And they seem to speak completely different languages. And one of the things that I think is amazing about DevOps is that it gives both IT geeks and company directors a shared language that they can speak, which helps them to understand the economics of software development basically. And it helps to put general good practices in cold economic terms. And I think when you can do that, it creates far better conversations in the boardroom about whether we take this approach or that approach. I think it’s much healthier. As it happens, for exactly the same reason, that’s why it becomes a marketing buzzword, because if you can do it internally at your organization, when IT folks are talking to senior managers, all the sales and vendors, everybody who sells anything that could vaguely have the word DevOps attached to it suddenly finds that they can do the same thing in their sales pictures and their marketing content. So that’s why both I think it’s amazing and also why I think it’s kind of being seen as a bit of a marketing buzzword. I mean, ultimately, it’s all rooted in the fact that these are general good, common sense ideas that actually work, but the message can be implemented differently and better or worse, depending on your motives.\nMS: So I would say that the DevOps is not only some kind of new technical trend that is just visible on the market, but it’s also some kind of mental change for the whole organization, that there is much more connection, or even alignment between the business and IT, correct?\nAY: Yeah, absolutely. And that’s exactly what I’m trying to get out with my mentoring stuff as well. I began to get frustrated. When I first started my company, I was basically saying “I’ll come and help you build your deployment pipeline” and I approached it very much from a technical perspective and developers and DBAs understood what I was talking about but the managers didn’t really get it, and I often found I was going into an organization and I would help the tech folks put together some great tech solutions for releasing changes reliably and regularly, but it’s always a human problem at the end of the day. Gerhard Weinberg has a famous quote: “No matter what they tell you, it’s always a people problem”. It was so true. I would go in and I would help somebody set up their deployment pipeline and then I’d realise: actually, you know what? As amazing as this is, I’m not sure how much benefit this organization is going to get out of it, because IT managers just don’t seem to understand how to manage projects in a way that’s going to work. I can give you the most amazing deployment pipeline in the world but if you’re still only planning to release once every 12 months, then OK, I may have made your life a little bit easier at that deployment time, but ultimately you’re not really going to see the benefits because the very first time you use this deployment pipeline is in six months time, by which point you’ve forgotten how to use it. It’s missing the whole point. The complexity of the deployment, quite apart from how you ship it to the end user, like the complexity of this code that you’re deploying means that your deployment is obviously going to fail. So I got frustrated with that, so I’ve been deliberately trying to not just help tech folks to build deployment pipelines, but actually come in at a slightly higher level and mentor the senior leadership to help them understand that a lot of the reasons why the organization is failing at delivering updates isn’t solely because the tech isn’t working. It’s because the tech are being forced into managing their projects in a way that fundamentally is well-suited to large, scary failures.\nKN: And I think you manage that very well, working for and helping Skyscanner develop the ability to deploy almost 100 times a day. Wow! Tell me something about that.\nAY: To be fair, that was a long time ago now.\nKN: A few years ago, right?\nAY: It was, and actually when I did that, that was very much at a technical level. I probably need to update my buyers. So that was back when I worked for Redgate, rather than DLM Consultants. I worked with a few folks at Redgate, including Grant Fritchey and Steve Jones, a group of five of us put together a training programme to help folks understand how to use the Redgate SQL Change Automation PowerShell cmdlets, which were brand new at the time. We had this new product at Redgate that would help people to deploy their stuff and I was a sales engineer working with Steve and Grant and a couple of others on how we can actually train people and coach them how to use it. So we put together this training programme and we delivered it in a few places and I’ve been speaking in my role as a sales engineer at Redgate to some of the folks at Skyscanner, and a few of them came over to this training class. So obviously we went to the training class and helped them to explain how it would work. We had like a whole bunch of whiteboarding sessions specifically with the folks at Skyscanner because they had some interesting technical problems, which I’m not sure how much I’m allowed to talk about them because probably NDA, so I’m not going to go into any detail. But they had some interesting scale issues, so we did a bunch of whiteboarding about how to put it together. And then they went and tried it and it worked really well. We had a webinar afterwards to talk through the implementation. And yeah, that design that we put together, I think it was at a precon at SQLSaturday Exeter many years ago, that became their deployment model for production. And that model that we put together on a whiteboard in Jurys Inn in Exeter became the model of how they manage their database deployments. And they were able to scale that up something crazy. And it wasn’t just 95 times a day. It was also because each one of those deployments doesn’t just go to one environment. It gets fanned out to like hundreds of databases. So yeah, it was a really, really interesting technical problem. And it was right at the beginning of SQL Change Automation. And it was also one of my first real experiences of consulting. I mean, I was still at Redgate and I was still a pre-sales engineer at the time, but\u0026hellip;\nKN: But you felt like working on the client-side, yeah? As a consultant.\nAY: Absolutely, after I’ve spent a couple of years at Redgate, there’s only so many Redgates SQL source control demos that one person can do. And I didn’t enjoy trotting out the same demo over and over, and over again. What I really enjoyed was saying: “Right, OK, that’s your nice, wonderful perfect world problem. Here’s my real-world complicated scenario. How are you going to make those tools work in this real-world scenario?”. And so it’s really fun to kind of think through all of that. And that definitely set me on the path to wanting to be more of a problem solver. And I enjoyed that an awful lot more than I ever enjoyed selling software licenses, which is ultimately the reason I quit Redgate.\nKN: OK, this is a very good moment. So tell me who said that: “Try quitting the best job of your life without any idea where your next pay check is coming from”.\nAY: I believe that was from one of my blog posts. I can’t remember but it sounds a bit like me. It might be on the About Me page on my blog.\nKN: Yeah, it’s you. It’s a quote from your website. It’s from a page about you, basically.\nAY: Yeah, Redgate was the best job I ever had. And I was a film star for a bit. I say “a film star”\u0026hellip; I walked in the back of a Harry Potter film. I’ve been to work for the day with Daniel Radcliffe and Rupert Grint and the rest of them. But yeah, Redgate was definitely the best job I ever had, until I decided to start working for myself.\nKN: Exactly, you decided to resign from Redgate in 2016, right? And started your own business.\nAY: Yeah, it was the day before the Brexit referendum. And obviously, I was nervous going into that, and if we’d voted to leave, it was like “well, what’s going to happen to the economy?”, so in many ways that was a “right, I’ve got to do it now, because I don’t know what the result is going to be tomorrow”. I actually expected the result\u0026hellip; Everybody else at Redgate was like “no, of course we’re gonna remain”. But that’s because they lived in one of the most pro-Remain cities in the UK. So I was like “I don’t know what’s going to happen but I’m blatantly not going to have the confidence to do this tomorrow, so I’m going to do it now”. So I handed in my resignation the day before Brexit, so that I couldn’t go back on it. And then I couldn’t go back on it.\nKN: What inspired you to take that step?\nAY: Well, I’ve been \u0026ldquo;umming\u0026rdquo; and \u0026ldquo;ahing\u0026rdquo; about it for a while. Over the last few years at Redgate, organizations like Skyscanner and various other organizations I got to work really, really closely with, more as a consultant, I really, really enjoyed that. As I worked my way up the ranks and sales at Redgate, obviously at Redgate you start as a salesperson answering the phone to people who want to buy SQL Compare and trying to convince them that they want to do something bigger than that. And over the course of several years, I worked my way up to being a pre-sales engineer, much more of a technical consultant and I was speaking at events and I was beginning to really make some friends and get to know people in the SQL community. And the more I did that, the less I took satisfaction out of selling licenses and the more I enjoyed solving problems. So my last couple of years at Redgate, I was more and more thinking “I really, really want to go and do this, some sort of consulting thing”. I recognized that most organizations don’t need a full-time Alex Yates. They need somebody to come and show them the right way, and then I need to go. So it felt to me like I can deliver the most value if I go to organizations to deliver training or consulting or whatever, or mentoring for a short period of time and then leave and go somewhere else. That to me felt like how I could deliver the most value to the world with the skills that I had. So it’s like “well, that’s what I want to go do”.\nKN: And also make your life happier? Because you need to be happy with what you’re doing at your work, on the professional side.\nAY: Yeah, absolutely. So it’s one of those things, in any job there will always be bits of it you don’t enjoy. Frankly, I’m running my own company right now, I hate accounting. I just can’t stand anything to do with it. Like, I wanna earn money from customers, invoice them, pay whatever tax I need to pay and then keep the rest of it. In my mind, I would love it if it was that simple, but it rarely is. I hate everything to do with it because law is one of those things that it doesn’t matter whether you agree or disagree with it, you end up just having to do it. And I just find it so boring. So for me now, accounting is the thing that I hate the most.\nKN: But do you have your own accountant?\nAY: I do but it’s not as simple as just having your own accountant because there will be questions about, well, “how should I do this, how should I do that?”. Because of the nature of the work I do\u0026hellip; The very first year I was in business, I didn’t have any UK clients. Every single customer I had to get on a plane for, which means that I was working with different customers in different countries, different currencies, so it wasn’t good enough just to understand how to manage all my finances based on UK law and UK taxes and VAT and stuff. I had to understand how that works internationally. And then I was also doing online training sessions that people could buy tickets for, and they could be buying tickets from anywhere in the world so there were various different business models I had and each of them just seemed to raise more questions about how I do this and how I do that. So just because you have an accountant doesn’t mean you don’t have any accountancy issues. And then there’s bookkeeping as well which is another whole part of this. Frankly, when I started the company, I didn’t even know what an accountant did or what a bookkeeper was. I have since learned\u0026hellip; My favourite thing that I used to do when I was doing my accounting and doing all my bookkeeping, getting all my receipts out and logging them and using my accounting software and putting in the tax for this and the tax for this, have you ever watched “Black Books”, series 1 episode 1, the one where Bernard is doing all of his accounts?\nKN: No.\nAY: It’s a hilarious British comedy with Dylan Moran and Bill Bailey. Dylan Moran is a bookshop owner and he has to do his accounts, and he feels roughly how I feel every time I have to do my accounts. So what I used to do is I used to put that episode on, put all the receipts out in front of me and pull myself a massive glass of red wine. And that was how I would do my accounts. Sooner or later that didn’t work very well, so at that point I hired a bookkeeper. But my point is that whatever job you do, whatever you’re doing, there will be bits of it that you don’t like. And for me the bit I don’t like is accounting but I tell you what, it’s a lot easier to do the bits of the job you don’t like when you’re doing it for yourself. Because when I was doing it for somebody else, and I had jobs that I didn’t like, I just found that really hard to drum up the motivation to do all the boring jobs. Whereas when you’re doing it for yourself, it’s like “well, I have something invested in this”. For me, there was just a big personal life difference, the amount that I enjoyed what I was doing and the amount that I was willing to put up with the bits I didn’t enjoy.\nMS: OK, can we move back to the very beginning? So the question is when and how did you start your journey with computers? When and how did it start?\nAY: When you say computers, do you mean coding?\nKN: Something bigger than the calculator.\nAY: Well, I enjoyed playing my Nintendo 64 as a teenager. I was pretty good at Golden Eye and Mario Kart, and Rogue Squadron (that was a good one). When I went for my initial Redgate interview, I didn’t know the first thing about tech. I was just looking for a job in sales because I graduated with a drama degree and didn’t have any skills and sales felt like a way I could earn a decent living. And they asked me to do a technical presentation as part of my interview. And the technical presentation was “Describe what happens when you type a web address into a web browser”. Kind of what actually happens at that point and I have to present that. And the feedback that I got afterwards was that I did very, very badly at that. I actually did quite well at the sales part of the interview which was like a roleplay negotiation call. Apparently, I did very well at that, but I was hopeless at the technical part. So I started my job at Redgate feeling like I’d fluked the interview. And despite the fact that I was so rubbish at tech, they’d taken a punt on me because apparently I was good at sales. And I find it ironic now that actually it turns out I really didn’t enjoy the sales part of the job and it was a tech stuff that I really enjoyed. I think it was at Redgate\u0026hellip; I’ll tell you what it was. It was a few months into me starting at Redgate, I started at Redgate at about the same time that Grant Fritchey started at Redgate. I mean, I didn’t know who he was. I mean, I barely knew what a computer was let alone knowing who was who in the SQL community at the time. But Grant had flown over to the Cambridge office and the sales team had grabbed a meeting with him, because most of the salespeople were graduates who had never met a software developer in their life. And it was an opportunity to kind of meet a DBA, because these are the people that we sell to, and talk to them and kind of understand who they are and what makes them tick, because that would hopefully help us to build up rapport with them in our sales calls. And somebody asked him a fairly direct question. Something along the lines of “how can you get a DBA to like you as a salesperson?” and he basically burst out laughing and said DBAs hate salespeople. He said “look, what can you teach a DBA about how to build a database? Like if you’ve got a DBA who has got decades worth of experience building this stuff and you, some graduate salesperson, never written any software in your life, you’re going to tell him how to source control and deploy his databases. Are you kidding me?!”, and he just laughed in our faces and I was like “yeah, that makes a lot of sense, why would anybody trust me?”. So what I took away from that was that if I actually wanted to sell anything, if I wanted any DBA to take me seriously, I needed to learn my stuff.\nKN: Technical stuff?\nAY: Yeah, exactly. So at that point I kind of made it my mission to make sure, like, I can’t be a better DBA than Grant Fritchey, no way, but I might be able to learn one specific niche better. Like, take everything else away, I’m never going to be the person you call to performance tune your databases. I’m probably not going to be the person you call to set up an amazing monitoring system. But I can learn one thing, I can just take one sliver and make sure I’m the expert at that particular niche. And if I can build up the ability to talk about that confidently and to know that little bit more than the DBA, then the DBA\u0026hellip; I’m never going to tell the DBA anything to do with query plans but I’ll be able to speak with authority on that piece. And if I can demonstrate that level of understanding, then maybe DBAs will listen to me on my sales calls when I am talking to them about various source control strategies. So that’s what I did, I made that my mission. It took me several years before I could speak and open my mouth without embarrassing myself, but that’s what I tried to do. I did an open university degree in my spare time, I set up a coding club at Redgate, I intentionally went out of my way to sit with the developers, rather than the salespeople at lunch breaks and things. And over the years, through osmosis, I became sales champion for various products, typically the source control deploy type products. And so I was attending stand-ups, and I was getting into the rhythm of how developers work, what makes them tick, and I was also handling the feedback process, so like, the people in your organization who typically speak to your customers the most are the salespeople, and in most organizations that’s a massively untapped resource, because the salespeople know what makes the customers tick. What do they like, what don’t they like, what sells it, what doesn’t sell it, what makes the customers really annoyed about it. So I took responsibility for managing that feedback process, kind of bringing all the feedback together from all the salespeople and then presenting it to the developers and saying “look, this is what sales are telling you”. So in the process of doing that, like all of those things combined together meant that I really, really had to learn the topic of database source control and deployment like the back of my hand. Because typically, our customers would come up with complicated technical things and in order to be able to explain it to the developers, I’d have to be able to understand it and then the developers would ask me questions or I need to have a conversation with the developers about “what if we did this, what if we did that?”.\nKN: And you wanted to lift up your skills in that area.\nAY: Exactly, and the more I did that, the more I enjoyed it.\nKN: So that’s a very good time to go to the next topic I wanted to ask you and talk about, which is MVP. So got the first award in 2017. I remember exactly, we were sitting in the venue at SQL Grillen in Germany when you got that “Wow” email, right?\nAY: Yeah, I’d actually received it one or two days earlier but it had gone to my junk folder. And I was about to delete it because it just looked like Microsoft spam. I was literally about to delete it. I knew I’d been nominated for MVP but because of the way the schedule was back then, I was expecting that if I was going to be accepted it would have been the previous month. So when I didn’t hear anything the previous month, I assumed I hadn’t been selected. Which is fine – it was like the first time I’d been put up by MVP and I’d only been nominated by one person and I definitely had and to some extent still do have an inferiority complex. I think all of us do. And I was like “well, clearly I’m not gonna be selected”. So I never really expected to be selected and when I didn’t receive the email, I thought “well, it’s nice to have been nominated, maybe if I keep plugging away in a couple of years, I’ll make it”. So I wasn’t expecting it at all. This email came in, I was just going through my junk, just clearing it out as you do periodically, and I saw this email, and I was like “holy cow, that’s amazing!”. And it was really, really cool to be at an event when I received it, because it’s a really, really special thing to receive that email, especially for the first time, but not many people are lucky enough to actually be at a SQL event when they get the news. It was lunchtime-ish at SQL Grillen. I think I had two sessions that day, because I did one session for SQL Grillen but on that same day I did another session for GROUP BY. And William Durkin who manages SQL Grillen, he was so kind, he was so great, in the middle of SQL Grillen he gave me a lift to his own house, so that I could present my GROUP BY session from William Durkin’s house.\nKN: So you presented from his house, OK.\nAY: Yeah, so if you look back on my GROUP BY sessions, I think it was the second one I did, “Getting CI right for databases”, something like that. If you watch that session, that’s on the day that I found out I was MVP. And I got the email in between these two sessions, and I was surrounded by like everybody at SQL Grillen, a whole bunch of the SQL family were there. You were there, Prathy was there, I think the first person I told was Eelco Drost, who now works for Data Masterminds with William.\nJust after get known about his MVP recognition KN: And we took a picture of that moment, right?\nAY: We did! We’ve not got a webcam but that picture is up on my wall. You, Kamil, you are on my wall. If you ever see me presenting, I’ve got a bunch of pictures behind, and I got that photo printed out and I’m looking at it right now. It’s the top one, which means sometimes, depending on the camera angle of my webcam, it gets trimmed off.\nKN: I was surprised at the beginning. Thank you very much, it’s a big pleasure. I will attach the picture to the post on the blog. It’s you, it’s me and it’s Prathy.\nAY: And you’re on my wall now, right next to my very first MVP certificate. Pride of place, top and centre of my wall behind my webcam.\nKN: Thank you!\nMS: What does MVP mean for you and what are the benefits that are most important for you from your perspective?\nAY: What does MVP mean for me? It’s an honour to be recognized for it. I mean, it does come with some obvious advantages. It certainly gives you a bunch of gravitas. When you’re talking about a topic, people tend to listen a bit more, which is nice. So there definitely have been advantages. I have had customers that I’ve worked with who either started out by searching for an MVP that lives near them that knows this topic or who have said when they were comparing me versus somebody else, it was the MVP that’s swung it. So yes, there have obviously been business benefits to it. And I think every single MVP would probably say the same thing. But mainly it’s just an honour to be recognized for the work that we do for the community. We don’t do it for that but it’s nice to receive it. And it is amazing. But the biggest benefit, to be honest, is MVP Summit for me. Up until COVID, the ability to go over to Seattle once a year and kind of see what’s going on at Microsoft and get the inside loop on what they’re thinking and get the ability to talk to them about the direction with various Microsoft products, that’s really, really, really cool.\nKN: And the opportunity to meet them in person. The Microsoft campus as well, that’s a great experience as well.\nAY: The opportunity to have a beer with them and rant with them about “we should do it this way, we should do it that way”, they’re quite fun conversations. Many a time it’ll be getting into the small hours of the morning in a bar somewhere in Seattle and you’ll see geeks arguing about\u0026hellip; Arguing is a strong word but debating the direction that Microsoft should be going on this topic or that topic. It’s a really fun time for a geek. That’s cool.\nMS: It’s great pleasure to meet someone in person but nowadays there is a pandemic situation so\u0026hellip;\nAY: Now it’s all done over video conference and to be honest that saves us all a bunch of money. I mean, it’s never quite the same to meet somebody over video conference as it is face to face, but I’m not too upset about that, to be honest. I did always have a conscience about the environmental impact of everybody flying over to Seattle, so doing it remotely suddenly makes me feel an awful lot better about that. So there are positives and downsides. Obviously it’s a lot less expensive, because I don’t have an employer to pay for it for me, I’m paying for it all out of my own pocket. Oh well, I’m paying for it all out of my company’s pocket. So yeah, doing it remotely saves a bunch of money and is much better for the environment. So pros and cons.\nMS: It’s affecting the conferences but it’s also affecting the local communities, so how do you think it will affect the local communities? Because at the moment we cannot have the normal meet-ups as it was a few months back, and how do you think it will go in the future?\nAY: I don’t know, I don’t have a crystal ball. What I would say is that one of the things I’ve seen a lot is local meet-ups meeting on Zoom or Teams or whatever other video conferencing tool they want to use. And in many ways that makes stuff more accessible and more inclusive. It’s a lot easier for a lot of people to attend. And I wonder whether there is going to be more of a democratization and wider distribution of on smaller online type things in the future, rather than big central conferences. I’ve always cared more about local events than events 3000 miles away. MVP Summit is a bit of a unique one because there isn’t anything like it anywhere else. And you’ve got various smaller meetings throughout the year but MVP Summit is a bit different. But as for other events, I’ve always preferred local, domestic events over big, international, everybody-get-on-a-plane events. And I wonder whether the shift to online is going to take that a step further where people need to travel less but can access more. Yeah, there are questions about losing kind of the hallway track and the face-to-face and the networking, the communication. I don’t have a much better answer to that than anybody else does at this stage. I think that people will get better at that. Whether they’ll get good enough at that to ever make in-person events redundant, I doubt it. But I hope that the effect of this is a bigger decentralization of events so there’s a shift to smaller, more local events with less travel than bigger, centralized, everybody in the world fly to the same city. I would like to see more smaller and local events, and online events.\nTo be continued\u0026hellip;\nUseful links Alex\u0026rsquo;s profiles \u0026amp; websites: Twitter | DLM Consultants | Speaking Mentors | Blog\nConferences: Data Relay | DataGrillen | GroupBy | SQLBits\n","date":"2020-10-30T08:14:33Z","image":"/uploads/2020/10/Alex-Yates-DLM-Consultants.jpg","permalink":"/2020/10/asf-034-alex-yates-interview-part-1/","title":"ASF 034: Alex Yates interview (part 1)"},{"content":"Yesterday was my last day of being Microsoft Most Valuable Professional. There are several cases where you may leave the MVP Program:\nbreaking MVP agreement by, for example, sharing confidential information lack of community activities required to be renewed resignation joining Microsoft In my case it was the last point – today (1st of September 2020), I am joining Microsoft as an Azure Technical Trainer in Worldwide Learning team.\nMy journey I have started my journey with IT many years ago by passing Microsoft Certified Professional exam in 2005. Since then I have been working on a few different positions including System Engineer, IT Specialist, Database Administrator and Independent Consultant. It was a bumpy road with many challenges, but with the help of others, I was able to move forward. In the beginning, my main focus was on technology, and I have been believing that technology is the key.\n1st milestone One of the milestones in my life was the challenge that we had during the studies with the Internet Service Provider (ISP) – ISP was blocking the possibilities of sharing the Internet within the flat that we have been renting. I have started to dig about the solutions of the issue and there were two options:\nuse some Linux-based device (router) with software increasing TTL (Time To Live) – quite safe option with many tutorials on the Internet how to do it more challenging option with installing Microsoft Internet Acceleration and Security (btw. all Azure firewalls GUI started here) software to implement a full-blown proxy solution – this option was much more challenging from hardware and software perspective As you can expect, I took a more challenging path with many failures during the implementation. However, I have learned that you have always at least two options, no matter what types of challenges you encounter during your life.\n2nd milestone The second milestone was when I have started my first permanent job with Metrosoft and the way how my manager (Jurek Połczyński) immersed myself into corporate life. It was the first time when I have travelled to the customer in the UK for the deployment of the software. You can imagine that it was extremely stressful. However, after successful deployment - satisfaction was also significant. I have learned that with most of the barriers are in our head, and with hard work you can achieve more than you expect.\n3rd milestone The third milestone was when I started working with ABB. As a part of the onboarding process, I had sessions with a great professional and mentor - Charley Hanania. He started the spark about the community and that our biggest gift is the ability to help others. In some way it is related to the next milestone: PASS Summit in 2012 in Seattle, WA – it was my first trip to the United States and I didn’t know what to expect there. Whenever I am travelling somewhere I have some expectation and picture how the place would look like etc., but each time when I am reaching my destination it is different than I expect. Don’t take that it is worse or better - it is just different. On the conference, I was attending both pre-confs and full conference (five days in total) and I was amazed by a warm welcome of the other attendees and also speakers. I am an introvert person, so pushing myself to talk to the strangers on the conference was an exhaustive experience. Even though I was on the other side of the globe, I felt that I can take with the other persons who have similar professional interests. Now I know it is called #SQLfamily ?. The conference itself was like drinking from fire-hose, so many different ideas, a huge amount of information, access to the Product Group (SQLCAT, etc.).\n4th milestone The next milestone was the first meeting of the Polish SQL Server User Group (now it is called Data Community). I have joined the meeting in Kraków, that was and still is led by Tomasz Libera. I felt that a group of professionals that have similar interests and also similar challenges in professional life. After a few sessions and encourages by Tomasz, I have decided to present my first session about HA/DR options within SQL Server. From today’s perspective, I can say that session was terrible, but without this step, I wouldn’t be in the place that I am today. I have been attending PLSSUG meetings frequently and polishing my presentation skills from time to time.\n100th meeting of Data Community Krakow chapter This led to the next milestone – being co-leader of PLSSUG Krakow group and helping others to succeed with many different goals (first presentation on a local group, on the conference, solving technical issues, etc.) The learning from that milestone was that a group of person can do more than a sum of individuals.\n5th milestone After some time I wanted to taste how the life of consultant looks like and thanks to Charley Hanania, I was able to work with clients around the globe with many challenges and opportunities (like being MCT Coach on the Microsoft Inspire conference in Las Vegas, NV). Very inspiring for me was visit in Museum of Flight in Seattle, WA, where I have seen many great examples of human inventions like Lockheed M-21 Blackbird with D-21B Drone, the first every built Boeing 747, Concorde, Apollo Command Module and many, many more.\nStatue of Lt. Col. Michael P. Anderson with a quote: \u0026ldquo;Dreams really do come true\u0026rdquo; After many hours of discussions with Charley, I have decided to apply for MVP and finally, in April 2020 I have been awarded.\nOn 1st of April 2020, I was awarded Microsoft MVP The lesson from that milestone is that you need to be careful about what you are dreaming - because it can happen.\n6th milestone Today, I am starting a new milestone in my life – joining Microsoft as an Azure Technical Trainer helping Microsoft’s customer to succeed using new technologies. I am very excited and at the moment I feel like having in my hand a ticket to the amusement park with many roller coasters and other attractions.\nOf course, achieving these milestones won’t be possible without the help of great people (like Charley Hanania, Łukasz Grala, Paweł Potasiński, Tomasz Libera, Kamil Nowiński, Mihail Meteev to name only a few). With the help of them, I have realized that the technical aspect is required, however, people are the key.\nSummary I find the very inspiring speech of President of United States of America, John F. Kennedy at the Rice University, especially the part: \u0026ldquo;We choose to go to the Moon in this decade and do the other things, not because they are easy, but because they are hard\u0026rdquo;\nLean back and enjoy the Azure I hope that this blog post is inspiring for you to start leaving your comfort zone and exploring new possibilities in your life. I am not leaving SQLPlayer blog nor Data Community, so expect new blog posts and virtual meetings in close future.\nDo you have a similar milestones situation in your life? If so, please share them in the comments section below!\n","date":"2020-09-01T21:10:42Z","image":"/uploads/2020/09/microsoft.jpg","permalink":"/2020/09/my-last-day-of-being-mvp/","title":"My last day of being MVP"},{"content":" Introduction Denis is a Data Architect. He has more than ten years of experience in software development, database design and performance tuning for cloud and on-premises solutions.\nIn 2010 Denis became Microsoft Data Platform MVP and maintain the MVP status till now.\nApart from his professional career, Denis actively participates in the development of Ukrainian SQL Community and hold the status of PASS Regional Mentor for Central and Eastern Europe.\nDue to the coronavirus outbreak, this talk has taken place online on 09 July 2020 (Thursday) using Zencastr platform.\nInterviewer: Michal Sadowski (T) \u0026amp; Kamil Nowinski (T).\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript MS: Hello, Denis, welcome to Ask SQL Family podcast.\nDR: Hi, Michal. Hi, Kamil.\nMS: So let\u0026rsquo;s start with the first question. What is your name and where do you live?\nDR: So my name is Denis and I live in Kyiv, Ukraine.\nMS: Okay, what you are doing for a living?\nDR: Yeah, honestly, I am doing a lot of stuff - about probably 50 per cent, this is performance troubleshooting, performance and issues troubleshooting. Other 50 percent is involvement in product development, architecture and all of this kind of stuff. But, you know, I can say this isn’t 50/50 all the time. It depends on the amount of client issues happening. The amount of releases we have this month and a lot of other factors, but probably doing different stuff. And mostly for SQL Server. But for the last year, I\u0026rsquo;m also invested in my own time into Postgres learning and into I am diving into Postgres issues and some architectural questions.\nKN: Which engine are working the most right now - SQL Server or Postgres?\nDR: SQL Server.\nKN: SQL Server, great! So what kind of troubleshooting are you facing? Generally.\nDR: Well, it\u0026rsquo;s pretty different all the time. I am not doing it now. You know, I\u0026rsquo;m not a big fan of approach when you have all your scripts set up and you\u0026rsquo;re on them all the time on the issue happening and doing all this. It quickly become boring to me. And right now, I am started with talking with client about the issue, how this happened, and when this happened, and trying to gather the initial information. And then I dive with scripts, extend events, sometimes Profiler, performance counters, collect the information particularly for this case. And then to investigate it. When I see that this case is similar to the previous one, this makes my job a lot easier because I know what the issue is and I\u0026rsquo;m going into it. But in probably 80 percent of time, the issue I am working with is different. And it requires a bit different approach. So this helps me to make my job faster. And it became fun because each issue is different. You investigate it in different ways and I feel that this is driving me. I really love this process and how it\u0026rsquo;s going on.\nKN: I still remember one of your sessions during the, probably it was during the SQLDay conference when you were talking about the deadlocks. So how often are you facing the deadlocks problems at your work?\nDR: Pretty often. Pretty often. Big deadlock issues are happening not that much. Yeah, but some of them can escalate, so we\u0026rsquo;re quick under heavy load. And about not five but about maybe four years ago for one big customer, we have the rate of deadlocks thousands per day, tenth thousands. And that was yeah, that was pretty big issue for them because the system quickly became slow.\nKN: Yeah, definitely.\nDR: Yeah, it was happening, not so fast. It was growing, growing and growing until that point when the issue do sufficient big effect on the system, And that is the point when I was involved.\nKN: So in that point, when you\u0026rsquo;re involved, what is the best solution or what is the best maybe in general approach how to resolve it and how to in the future avoid that situation.\nDR: For that case, the approach was to switch the transaction isolation level for those particular queries involved in the deadlocks. We switched them to snapshot isolation on application-level. So for this issue, I worked through with the development team and this is often what I do. I work with the development team, with services department team, with support team. On my position, I work in with all departments and different stuff. And I really love these because I love this interaction. Also, I am interacted, I\u0026rsquo;m working with clients a lot. This is pretty interesting and pretty entertaining. And for this particular case, we do these change on an application level. The product is pretty big and complex and it\u0026rsquo;s very risky to go with read committed snapshot, because you never know. Some parts of the products will change their business logic after this change. In this case, snapshot isolation is much safer because you can set it on, you can hold all database with read committed like it was previously, but for several sessions, particular sessions, you can set them in snapshot isolation. And this is what we\u0026rsquo;re doing in the product. On the start application check if snapshot isolation is enabled. If it is enabled, it started these few queries in snapshot isolation. So all other approaches we share we\u0026rsquo;re thinking about did not work. I found this pretty quickly because the queries involved into deadlocks had a lot of variety. And we can\u0026rsquo;t we preserve them. Scan direction forward or backward. Because for some cases, in the query plan, it will be better to do it forward for some cases it will be better doing backward and we can\u0026rsquo;t really avoid deadlock in that case without removing shared-lock and it is what we did. Sometimes I see deadlocks when I do troubleshooting for other issues like timeouts or heavy queries. I am just checking the system health session, if there are any deadlocks there. Sometimes there are few, sometimes there are thousands, hundreds, and this is the point when I started investigation again for that deadlocks.\nMS: I imagine that solving the deadlock issues is not on the right to purely technical skills that you need to figure out which query it is, but it\u0026rsquo;s also related to learning developers\u0026rsquo; how to write proper code. And how do you find it with learning the others how to make this stuff working correctly?\nDR: You know, our developers are pretty experienced, but there is a factor when you write business logic, you really can\u0026rsquo;t, especially when you\u0026rsquo;re using tools like Entity Framework or ORM systems, it\u0026rsquo;s almost impossible to Predict outcome sequel query. And it\u0026rsquo;s impossible completely to predict if there will be a deadlock or not with some other query. Well, it\u0026rsquo;s pretty easy to understand that blocking tables in the one transaction, different order will produce a deadlock and this is a situation which we definitely avoiding. But more complex deadlocks are really, really heavy to predict and working on that deadlocks is mostly reactive, because you can\u0026rsquo;t do this proactively.\nMS: OK, so as I understood correctly, you\u0026rsquo;re always like to connect both worlds: I mean, business world and also the developers world that the product, the queries or the stored procedure that are being produced, they are working from the perspective of the business they are fulfilling the needs, but also from the technical perspective, they are just behaving as expected.\nDR: Yeah, we just admit this is a situation which can happen. We just need to react too quickly and do what we can. And also what we did, actually it was done in the product from the beginning, we have resiliency for the connection if the query was timed out or killed like deadlock victim, we just re-run this query one more time and we do this several times. So usually single deadlock is not an issue for us. we definitely can look into it because it may make the workload more efficient. But generally, the issue for our product is a lot of deadlocks. If this happens this is issue, this is visible for a client. If this is single deadlocks or few deadlocks, it\u0026rsquo;s fine., we can handle that.\nMS: You have described that you have a lot of experience from the SQL Server perspective from the SQL Server internals, you have delivered a lot of session about the deadlock around the world. And I have the question how the PostgreSQL world is different comparing to the SQL Server or is it exactly the same?\nDR: It\u0026rsquo;s very different, and, you know, the difference is not about syntax, but about the internals. I can\u0026rsquo;t say that SQL Server is better than Postgres or Postgres is better than SQL Server when a dive into the internals of these two engines you understand that they both have their benefits and the drawbacks. So, for example, in case of deadlocks. Usually when you write the same application, Postgres and SQL Server, you will have less deadlocks in Postgres because you don\u0026rsquo;t have the shared locks, but all other possibilities are still there.\nKN: So there\u0026rsquo;s no clear answer: which one is better?\nDR: I believe the better answer is to understand the internals of the database engines which you use, and that will give you the clue to avoid the deadlocks and or other performance issues. So, in the case of deadlocks, Postgres is better.\nKN: One more quick question: when it comes to the database project or managing the code of the database. So for SQL server, we have something like SSDT, which is SQL Server Data Tools, do you have something similar in Postgres world?\nDR: Honestly, I don\u0026rsquo;t know, and I don\u0026rsquo;t use these database projects honestly. I\u0026rsquo;m a big fan of plain old scripts. That\u0026rsquo;s it.\nKN: So incremental deployment, right.\nDR: Yeah.\nKN:OK, cool.\nDR: This works with all engines.\nKN: Exactly. That is the point. OK, Let\u0026rsquo;s go back to the general questions. I would like to know when did you start your journey in I.T. or with computer in general.\nDR: With computer, definitely it was probably in school. I think sixth or seventh form and I was started with ZX Spectrum. Probably the name of computer was Sinclair 128. And I really, I loved it. My friends had these 8-bits gaming consoles, but Spectrum was a bit bigger than that. Well, almost all the time I played games about 95 percent probably and five percent of time I tried BASIC. There was a book with the computer and there were some programs I tried to write code using them, learn something, that was pretty interesting. I can\u0026rsquo;t say that I love that but it was interesting. It was very different from gaming.\nKN: Definitely. I remember the BASIC I remember BASIC language. I started with the same many years ago.\nMS: I guess everyone from our years remembers, I started with Commodore, so I would say it\u0026rsquo;s like closest competitor\nKN: Commodore was the closest competitor to the Atari, I think. Okay. And from the professional perspective, working in I.T.: how did you start and when?\nDR: My first company was the DB Best. So this is a company focused on database products and at times that was almost on database products. There was a product which called SSMA (SQL Server Migration Assistant), and that times there was no Microsoft product to support migrations to SQL Server. There were products which we were supporting migration from everything to everything. Yes, big vendors like Oracle had their own products, but there was nothing for SQL Server, and this is when DB Best came with the product and sell it to Microsoft. So I came to DB Best after that, the company was very successful and they were developing this SSMA product for Microsoft. There were a lot of very good, very experienced developers, I know that a lot of them came to Microsoft after that. And what started in that company as a young developer in 2006, that was actually five, five year the last year in university here of the fifth year. I combine work part-time and study. And that was pretty good and actually that was the point where databases became my hobby. I was software developer, honestly, and in DB Best, I started my position as engineering in testing. And I was involved into the test framework development. Well, that was pretty good for the start but I wanted more, I wanted to work with the software, some products, to create a product from scratch. And then after a year in 2007, I came to another company, Softline, and my team leader was my university teacher, Volodymyr Leshchynskyi, and he became my mentor for probably for the next a lot of years. We came through several companies and end up our journey company, DCT. He was the director of the company and I was, at the end of my journey with that company, I was head of Cloud Development Department.\nMS: And everything was purely SQL Server, right?\nDR: No, no. I was started as a developer and was growing this expertise as a primary one for a long time.\nMS: So with which technology did you work?\nDR: .NET and ASP.NET particular. Those days we had developed a lot of product, but one of the biggest one was Inetgiant. That was a big advertisement website like OLX or Craigslist. We\u0026rsquo;ve not been so big as Craigslist, but we were in top 10 U.S. advertisement websites. So with max, we have 300 000 unique users coming to our site per day, that was pretty big load, I remember like when we ordered big servers, with a lot of RAM with a lot of CPUs used to handle that load. That was pretty good product where I\u0026rsquo;ve grown my database expertise a lot. Volodymyr knows me from university years because he was my mentor on my diploma, so when I came to the Softline, Volodymyr said: Okay, you\u0026rsquo;re coming from DB Best, probably your expertise in this company will be databases, start growing it. Okay. That was pretty interesting to me and I started growing this expertise and working as a web developer but database were my hobby and I grew and grew with this expertise, more and more. And each day, each year, the database world become bigger for me until at some year in 2015, I completely left application .NET Development and switched to databases.\nKN: In 2015?\nDR: 2015. Yeah.\nKN: But let\u0026rsquo;s back in time a little bit. What happened in 2010.?\nMS: We are just recording it on the beginning of July, So first congratulations for the anniversary. It\u0026rsquo;s 10 years and the question goes to the how it started that you\u0026rsquo;ll have a lot of experience, a very broad experience, not only with SQL Sever however in 2010 you have been awarded by the MVP. So how it happened that you have the MVP in 2010 and what was the journey about that?\nDR: Yeah, that was really interesting journey and very good. So in our company, we have a culture of learning. When I came to Softline and then through other companies, Volodymyr, my mentor make this happening - he created this culture of learning in the company. I remember we were young in 2007-8, he gives like in university, he gave us some tasks to prepare some session and give this session for your co-workers. That was pretty good experience for me. I started speaking, at the internal community, I am started doing some sessions, small sessions about SQL Server, I was listening to guys when they do other sessions, and Volodymyr tried to find the strong points for everyone and gave them the topics to prepare. So in our company, we were like MVPs on separate technology. That was pretty interesting. My was databases and my was SQL Server. After that we have big community in Kharkiv, probably this was the biggest and the oldest user group in Ukraine called Uneta, probably each month to have one or two sessions there and starting from maybe starting even from my university years. I started go into that sessions and that was like, wow, to me, that or something new that or something interesting. That was not university lecturer format, and there were very good speakers. They were coming from Microsoft, from other companies. That was a lot of knowledge to me. I really loved that and thinking that one day I will be also speaking there. And in 2009, Volodymyr asked me, do you want to speak on Uneta? Of course I wanted. So I prepared the session. You know, I remember that session. The session was about types of indexes in SQL Server. In 2009, yes. All of them are B-trees under the hood. But anyway, that was pretty interesting.\nKN: It still is.\nDR: Yeah. I compare my photos from that session and the latest session. Wow. That\u0026rsquo;s so big difference. You know, I remember that how I can\u0026rsquo;t start saying I was I was not very loud, people did not hear me, they asked to make me speak louder. That was terrible. But anyway\u0026hellip;\nKN: It is normal, you are nervous.\nDR: Yeah. I was nervous very much I was scared.\nMS: And I was also called learning. So every time you deliver the session the next session is better comparing to the previous one.\nDR: Yeah. But after session I feel l was not so bad. It was pretty, pretty good. I thought that. But now I\u0026rsquo;m thinking that it was pretty bad, for that time it was good to me. I started speaking more and more. I started blogging and you know those days in which we know that we stuck with Microsoft technologies. Actually, we loved what just was happening there. What was happening was .NET, what was happening with SQL Server, SQL Server 2008 was a very, very good release. I remember that, WOW, it has a lot of solid features, which were really good. And we knew that there was an MVP community. I read a lot of blogs that time. I know all these guys in MVP community, I know that a lot of them are MVPs, and we have been thinking that, WOW, this is this probably very cool to be an MVP, to be part of this club. But this is a very interesting point that we did not know what is happening inside or did this or this program is given to me. But we thought that it is pretty cool. So, was started, we already did some activities like speaking. I was started blogging that days where we were set this goal to become an MVP and started to force and do all these to become an MVP. And I have a partner here, partner in crime here, [Sergii [Lutai]](https://www.linkedin.com/in/lutai/) So Sergii started speaking at Uneta probably the year before me. And he was my partner in crime because when, for example, when Microsoft asked us to go on some event, we were go in together sometimes when there were a lot of work, we prepared our sessions in the train. It was very fun. And we became very good friends during all this stuff. So when he was preparing the session, he told this session to me, when I was preparing my session I spoke to him. I was giving this session to him first, giving feedback to each other. And we got the MVP probably almost at the same time. He became Silverlight MVP, I became SQL Server MVP. So we bought a beer, we have some small party in the office. That was pretty cool, I remember that, I was very happy.\nMS: What do you think about this MVP program compared to the program when you started to be MVP, so like 10 years ago and the MVP program today. Is it changing or is it changed or is to the same as it was before?\nDR: It is changing, but it\u0026rsquo;s still I think that it\u0026rsquo;s still pretty cool. And, you know, this is not the end of the story, I should admit that the times I was pretty young and stupid, particularly in first years, because when I became MVP, first, probably three years, were very hard to me and I really wanted to quit.\nMS: Why it was hard for you?\nDR: I felt when I came to [MVP] Summit, It was pretty cool, this is my first time in the U.S. and I have almost missed the plane that goes on.\nMS: So it was some kind of the impostor syndrome?\nDR: Yeah, I saw all these guys, who are writing the blog posts and books which I am learning from. And I said, what the f**k I am doing here? I am not that good, that\u0026rsquo;s pretty hard to me because you know that like for actors, there are two approaches [[Konstantin] Stanislavski](https://en.wikipedia.org/wiki/Konstantin_Stanislavski) and [[Mikhail] Chekhov](https://en.wikipedia.org/wiki/Michael_Chekhov). Stanislavski forced you to feel the role, to feel that torsion emotions, feelings and play from this point. And Chekhov is from more from external stuff like if you are playing a driver, you should wear the hat, you should wear them costume of driver, you should sit in the car and play from this perspective from external things. I felt like I came to MVP not from internal impulse but from external pretending. And I did not feel like an MVP that days. So I am started evolving and I am started to improve my skills and all this stuff and it became much better. But honestly, I feel myself, so I get an MVP in 2010, I have done a lot of sessions about SQL Server 2008 in the next few years. But I believe the main point when I become that, okay, this is fine, I love this and this is my place. That was 2013 when I moved from Kharkiv to Kyiv changed my life a lot. That was very big point to me.\nMS: So this movement from Kharkiv to Kyiv started that you start to feel like MVP, not like the person that is aiming to be an MVP or it was something else, that was right to that.\nDR: Yeah, I believe at that point I have that kind of impostor syndrome and all kind of these. I feel myself like I tied into a small community in Kharkov and I wanted to try my myself in something bigger.\nMS: You are one of the organizers of the one of the biggest SQL Saturday in Europe. Europe, it\u0026rsquo;s the SQL Saturday in Kyiv. So how did it start?\nDR: Yeah. We will come to this shortly. I will tell the story how I moved to Kyiv. That was pretty interesting. So in 2012, there were a lot of things happening, but this product Inetgian was going down because of Google regulations and we completely lost it at some time. And starting from that date, my department, web development was becoming smaller and smaller because we have less projects and so on and so on. So at the end of 2012, I felt that I am not moving the company forward, but I am probably slowing it down. I was thinking that I need to change something and probably move to another company. Probably that was very good decision because this was not only about myself but about the company as well. And I feel that my boss has also taken me up, but it\u0026rsquo;s pretty hard for him too. Hopefully, another department was grown. Yeah. And it was pretty good the days, so I was pretty safe to go. I went to sleep and I thought, where can I continue my career in 2013? And honestly, two companies came to my mind. First one is the Frayman Group in Kyiv, there were lot of my friends and very good, no not very good, but, you know, exceptional experts working there. A lot of them came from Microsoft, Microsoft Innovation Center, from other big companies, and particularly there were Konstantin Kosinsky, first Microsoft MVP in Ukraine for SQL Server. And that year he moved to, this is a different story, but I went to his trainings to improve my SQL Server skills when I was very young. And here he was like a prototype for me, like a solid MVP, big expert in SQL Server. And I tried to become like him.\nKN: You had very good mentors and gurus in the SQL Server.\nDR: Yeah, he was really guru. And second company was Kaspersky because they have very good database expertise. I was thinking about it. And in the morning I wake up from the call Viktor Shatokhin from The Frayman Group called me and said, Hey Denis, how are you? Don\u0026rsquo;t you want to work with The Frayman Group? How this is happening?\nMS: Let me think.\nDR: Yeah. I asked for one day talk with my boss in DCT, he was fine. He was like, yeah, well we moved for a lot of years for only he was disappointed a bit, but he was fine and let me go. So I talked with Viktor and you know, in a few weeks I went to Kiyv and started my work in The Frayman Group. And that was the interesting story. Konstantin Kosinsky moved to Microsoft and I came to his position, particularly took the database expertise in company. And I was started with deadlocks because Konstantin went probably three, four months ago and there are some issues which needed to be fixed pretty urgently. And in that company in Frayman Group, I get my evaluation of my technical skills and I\u0026rsquo;m alleviate this impostor syndrome, because I feel: Okay, well, right here in new company, big company with serious product, very experienced people, the best people probably in Kyiv that time, I am getting very good. That was a changing point for me. And starting from that time, I feel that my sessions also became much better. Probably the first successful session they get in 2013 at Microsoft SWIT conference. I was talking about SQL Server 2012. But honestly, I when I started preparing this session, I thought that this session is not for me. I don\u0026rsquo;t like it. And I might actually do SQL Server deep dive session, instead of that, I called the SQL Server 2012 deep dive to satisfy the conference plot. But honestly, it was about all the layers inside SQL Server. When I came to the conference, there was a big rush there, a big mess. Organizers put me in this smaller room about 100 or 200 attendees, but there was too many attendees who wanted to see my session, that they were crowded, that they start arguing. They argued to organizers in Twitter and my session was urgently moved to the biggest hall. And it was completely packed. WOW, I was very honored and pleased with that. And I did my best to do this session and this session became the best on the conference and after that I like I validated myself as a technical specialist, and, you know, as an MVP, I also validated myself for that ok, probably, I am fine staying with this community, nobody will argue about this. And I really love the community. That\u0026rsquo;s it\u0026rsquo;s definitely changing its evolving, but it\u0026rsquo;s very good. And I love to see you guys there.\nKN: Thank you. Yeah, exactly. So it was 2013, right?\nDR: Yeah.\nKN: When exactly did you start SQL Saturday, organizing SQL Saturday?\nDR: Yes, SQL Saturday were started the bit early, in 2012 in Kharkov. The first Ukrainian SQL Saturday happened in Kharkov.\nKN: And it was organized by you or by the team?\nDR: Yeah, yeah, probably before I started to be an MVP. There is a guy in our community, Andrey Korshikov. Do you know him?\nMS: Yeah.\nDR: So Andrey Korshikov is really a father of SQL community in post-USSR countries region. I think he became an MVP earlier than me and I invited him to came to Kharkiv to Uneta to give a session. He came to Kharkov. He invited me to go Karsnodar. I went to Karsnodar and that was the first and the last time when I forgot my laptop at home. Wow, that was\u0026hellip; Fortunately I saved my session, slides to the OneDrive. But that was pretty hard to prepare a new laptop there and do all this kinda stuff.\nMS: I guess it was, what, one of the most challenging technical incidents that you had in your life?\nDR: Yeah, after that, I make myself a note that never forget laptop on your session. This never happened again. I have some other speakers notes that was the first note.\nMS: But I remember that in Katowice, SQL Saturday Katowice, you had some issues with the charger to the laptop. So you had the laptop, however, without the charger.\nDR: Yeah. That was the second one. We bought the charger in the supermarket, but it was not powerful enough to power a [Lenovo] W530. That\u0026rsquo;s pretty old and the big ThinkPad laptop, well it was charging when I shut down the P.C., but when I open it, it can\u0026rsquo;t charge. So I charge that before the session started and it was not enough to complete this session, unfortunately.\nKN: Yeah, I remember that situation where you\u0026rsquo;re presenting an almost one hour and I think the battery was enough for that session. Yeah. And also, I remember during the MVP summit when we were sitting together next to each other and I\u0026rsquo;ve seen your laptop it\u0026rsquo;s pretty big and very powerful, right? Probably the same.\nDR: I\u0026rsquo;m going to change it this year.\nMS: Did you observe any difference comparing the conferences that are in Eastern Europe or Central Europe, comparing to the U.S. or the Western Europe?\nDR: Almost, on U.S. I was only on MVP and PASS Summit, I would say that PASS Summit is pretty similar to SQL Bits in terms of attraction, sessions, entertainment. I don\u0026rsquo;t know that this is just my feeling, but it\u0026rsquo;s more like West Europe and the U.S. and East Europe. Yeah, because SQL Day is definitely different from SQL Bits. And this is just my feeling, my opinion, but I love SQL Day more.\nMS: Thank you.\nDR: If this is some kind of spirit, maybe. But you know, you come to give Slovenia, Krakow, Wroclaw and when you come down to Netherlands conference, Belgium conference SQL Bits, you feel the difference. I can\u0026rsquo;t say that something is bad or good, it just different. I am enjoying both kind of conferences, but probably because I am from Eastern Europe, it\u0026rsquo;s spirit is closer to me. So that\u0026rsquo;s it.\nKN: How would you encourage people to go to visit Ukraine for the conferences, for technical conferences?\nDR: Oh, that\u0026rsquo;s a lot. I\u0026rsquo;m just describing what is here in Kyiv. We have pretty good prices for hotels, for food, for attractions. And we have pretty good Ukraine cousin.\nKN: Oh, definitely.\nDR: The Kyiv city is very old and it has a lot of historical places, sights, and the time for SQL Saturday May actually the end of May is the best possible time to visit Ukraine. It\u0026rsquo;s not very hot and it\u0026rsquo;s definitely not cold. It\u0026rsquo;s warm. The visit card of Kiev is Chestnut Tree. And during this time, these chestnut trees have flowers. And it\u0026rsquo;s pretty unique experience. You know, when you walk down these historical streets, I see all these sides, churches, monuments, the Golden Gate of Kyiv from the Kyiv Rus times, you feel the spirit of this city and it\u0026rsquo;s really unique. I am just trying to describe all these to the visitors. Those who loves this tell this story to others. And I believe the Kyiv is too, very attractive now.\nKN: Yeah, definitely. Definitely.\nMS: We can confirm, we had the pleasure to be in Kyiv on SQL Saturday with Kamil so we can confirm that. It\u0026rsquo;s really nice place to go, especially for the SQL Saturday.\nDR: Thank you. I really enjoyed having you in Kiev.\nKN: We can\u0026rsquo;t touch our picture when we are sitting and drinking a beer. Very good beer, by the way.\nDR: Yeah, I agree. Oh, honestly, that\u0026rsquo;s a beauty. But that brewery was closed. It\u0026rsquo;s on there next year. We wanted to involve them to the sponsorship again, but they said, okay, guys, we\u0026rsquo;re sorry, we\u0026rsquo;re closed, but we have other pretty good beer. And if you go back to SQL Saturday, I feel that I am moving to a different question, but we became friends with Andrey Korshikov and he was the first from our community who learned about PASS. This organization registered his Krasnodar user group as a PASS chapter and went to the PASS Summit in 2011, probably, and tell me all this story and encourage me in Kharkiv and Konstantin Kosinsky in Kyiv to create the SQL user group registered as a PASS chapter and go to the PASS Summit. And this is what we did. We made few sessions, but again, I was pretty young and stupid and I had a lot of ambitious. Right now, I feel that it\u0026rsquo;s, well, maybe it\u0026rsquo;s good to achieve some goals, but definitely is not good for yourself. I didn\u0026rsquo;t feel complete in that time. Yeah, I feel like I\u0026rsquo;m doing this, this, this. I will be so cool. But it is not. So then I started this Kharkiv user group we made one meeting and I still thought that was stupid. We have big user group Uneta here, What the f**k? - I don\u0026rsquo;t want to run another one, I would rather support Uneta and my guys who run this user group. So this will be much better. And now when I came to Kharkiv every time I make a session on the Uneta user group. It\u0026rsquo;s not that big these days, but I have this old feeling that this is the user group where I was grown as a specialist, I learned a lot from the speakers of Uneta that time. But anyway, I organized SQL Saturday in Kharkiv and a lot of foreign speakers came to this SQL Saturday. And this is the point where MVP community and SQL (Data Platform) community interacts. Because when I went to MVP Summit in 2011, I meet a lot of MVPs and particular a lot of MVPs our region and we started talking. And next year, when I started SQL Server conference, SQL Saturday conference, I start writing to all of them, probable about 50 or 40 percent agreed to come. But that was more than enough. We had three tracks. We had a lot of foreign speakers, probably 70 percent of foreign speakers. By the way, this is the time when I met [Mihail [Mateev]](https://www.linkedin.com/in/mmateev/) and [Niko [Neugebauer]](https://www.linkedin.com/in/webcaravela/). So these three guys: Mihail Mateev, Niko Neugebauer and Andrey Korshikov actually go to the community guys who involved me to SQL/Data Platform community and who teach me everything I know right now. But probably right now I evolved and get some my knowledge, but that times they were my mentors and my teachers in the community. Mihail submit the session because this is what he is doing - he saw that new SQL Saturday, he submitted here an Niko came because he was PASS evangelist that years and Andrey came because we already know each other and that was a really unique experience. I organized SQL Saturday for the first time we had more than 100 attendees. It was very big success. We had a lot of positive feedback. And I saw the attendees at the conference - they really enjoyed it. After that, we had a beer that was probably the first post to the #sqlbeer hashtag there, but it was Niko, Andry, Mihail and me. We discussed the community, the PASS there all of PASS the community. All these questions about the conferences. Wow, that was pretty good. I learned a lot about the community, about the conferences, organization. Before that, well we had a conference Uneta Plus one year before. And I was involved, I was in the organization team, so I knew something about conference organization, but SQL Saturday Kharkiv was the first where, I was involved by myself.\nKN: And then the second one I was in Kyiv, right? When you moved to the Kyiv.\nDR: Yeah, but that was not my conference. You know, I was involved as an organizer, but I was more like a mentor. That conference was done by Konstantin Khomyakov, my good friend. And he was also working in Frayman Group. I met him on PASS Summit in 2012. At that times, Konstantin Kosinsky went to Microsoft. And he left Kyiv user group, the Konstantin Khomyakov. We talked with him about SQL Saturdays, so he was going to organize SQL Saturday in May in Kyiv. So when I went to Kyiv probably first day, we drink some drinks with Konstantin. Well maybe not some drinks, but we drink a lot to celebrate. And he told me the story, that\u0026rsquo;s different. That\u0026rsquo;s personal. But you know he is still my good friend, despite the fact that he\u0026rsquo;s living in Australia. I really miss him because he became an MVP few years in 2014 or 2015 he became an MVP. He came to few Summits. But now I haven\u0026rsquo;t seen him for ages. So at that time, he was a group leader and I was helping him to build the user group and SQL Saturday from my experience. That times I remember that we started from the sessions in our office, in the office of The Frayman Group company, and there are three, five people, ten people coming to our events on the on one year probably we have both 20 or 30 people who did not able to fit the room. That was what I remember. That was Vitaly Popovych session about range locks in SQL Server. It was pretty good after the time was started to find new place for our group. But SQL Saturday Kyiv was done by Konstantin. He was running the user group and I was helping him. And honestly, that was a pretty unique and good experience for me. This is, this was the principle which I built into user group and SQL Saturday after that in years. In this place, when you need a community to grow, when you need to do something for the community, you put your ambitions behind that. That was a pretty good moment for myself, for my personality, because I feel that I really love doing that. And the community win from these because we\u0026rsquo;re moving together but not separately, we\u0026rsquo;re not running different user groups or competing with each other, but together we\u0026rsquo;ve made much more than we will be able to deliver separately.\nMS: And also, it looks like that at the very beginning of your journey, you had a mentor. And now you are the mentor for the others.\nDR: Yes, that\u0026rsquo;s true.\nKN: And also, it looks like there is plenty of work, plenty of efforts you made during the all the years, at least 10 years, last 10 years, so I\u0026rsquo;m just wondering, you know, how would the priorities have been changed during those years and how the work-life balance works for you?\nDR: A lot of things changed and particularly user group right now is a big user of group activity right now. Is it be different from 10 years ago because conferences that days, there were not a lot of conferences, user groups in Ukraine particularly, but these days we have a lot of paid conference with a very good content, very good speakers, and it\u0026rsquo;s very hard to compete with that in user group. User group became a niche product. That days, it was main product, it was off as a main product probably in 2015, 2016. That was main product after that became less and less. Right now is a more niche product in Ukraine. Priorities changed a lot. You know, since that time I was married, my kid was born. I love him very much. I can\u0026rsquo;t say that a lot of things changed in this era. I still love user group activity and community. But yeah, it is different and last year I feel that I am this is this is personal, but I started to seek more and more and I feel that these have affect on my sessions as well. Each year, starting probably from 2017, it became more and more hard to me to deliver a good session, to organize a good conference. I think more and more efforts on this and this is what I did not enjoy it, enjoy it, too. Because sometimes. I started to do not very good sessions. Well, not completely bad, but not very good. I feel I can give myself a feedback. What was good? What was bad? What I can improve? What I can\u0026rsquo;t improve?\nKN: I\u0026rsquo;m not sure about you, Michał, but I didn\u0026rsquo;t notice that.\nMS: Yeah, I had the same feeling. I just had a feeling that you have too high you set too high expectations for that, that you would like to have each session that the next session is like 50 percent better comparing to the previous one. And you are at the moment top notch specialist, so it\u0026rsquo;s like impossible to have better session.\nKN: Exactly. Sometimes you need to just set the baseline, you know, very high-level baseline. But still it\u0026rsquo;s a baseline. Yeah.\nDR: Thank you. Thank you. But I feel that this is. This is myself. But I put myself under this pressure. So this year before 2020, I decided to not completely quit, but rapidly decreased the amount of sessions. So this year I did not came, probably you noticed that I didn\u0026rsquo;t came to SQL Day. This is nothing personal, I love this conference. I didn\u0026rsquo;t come to Belgium conference Data Mind Connect. I did not come at the conference fwDays in Ukraine, where I usually coming. I spoke a lot for several years, I will skip this year to have some rest, and avoid this pressure. And you know that before the New Year, I quit as a leader of Kyiv user group. So Ukrainian data community right now, led by Yevhen Nedashkivskyi, you know Yevhen.\nKN: Yeah\nDR: He was part of our organization team, he done a great job. And right now he\u0026rsquo;s leading a user group. I am still in the team. Yeah. We have four members of our community: Yevhen Nedashkivskyi, Eugene Polonichko, Mykola Pobyivovk and me. Yevhen is the main guy right now. As for SQL Saturday my last SQL Saturday was this year it wasn\u0026rsquo;t happening because of coronavirus. But next year, Eugene Polonichko will take this responsibility and became the main organizer, probably.\nMS: But I hope that you are still a mentor.\nDR: Sure. I will still be in this team, in the team anyway. And also, I quit from being PASS Regional Mentor this year, last year, actually. Before the New Year. Yeah.\nMS: So maybe it will be some kind of the vacations or holidays for like one or two years and then hopefully you go back to the community because you have a lot of experience that you can share with the others.\nDR: Yeah. Yeah, that is true. I have this kind of rest. I would say I\u0026rsquo;m going out from pressure and you know, right now I have I feel empowered to speak this autumn at some conferences and I will return next year and submit for a SQLDay, if you will consider my session of course. Despite that fact I missed this year, I will be very honoured and happy.\nMS: Are always welcome on the SQL day conference. In case we are discussing the future, how do you feel about the roles like DBA role, how it will evolve in the next few years because the world is changing, I\u0026rsquo;m not saying on about the pandemic, about the coronavirus, but much more about the technology. So from your perspective, how do you feel about the evolvement of the roles?\nDR: Well, that\u0026rsquo;s pretty interesting because I don\u0026rsquo;t like to make a forecast. Usually this forecast and some of the areas are not very good in 5, 10 years Let\u0026rsquo;s try. I predict the DBA role will not change a lot. Well, when I think about these distinguish between database developer and DBA, I know a lot of DBAs I\u0026rsquo;ve worked with very good, exceptional DBAs. And as for me, I am more kind of a developer, I have DBA and developer skills which I use in my work. But I came from the developer world. And DBA for me is not only about technologies and operational work, it\u0026rsquo;s some kind of mindset when I interviewed DBAs in our company or developers, I believe the DBA is more of a mindset that everything should be controlled, you should take care about your server. So these and that was done, and this is systematic approach. I have a checklist, DBA checklist to setup this and that they should be in this area. This should be balanced and so on. And so. So the developers are different in this area.\nKN: Are you trying to say that developers don\u0026rsquo;t care?\nDR: We don\u0026rsquo;t touch politics.\nKN: This sounds like a political answer.\nDR: Yeah, this. This is some kind, This is what I see. This may be my personal bias. But anyway, when you work with a problem so you have performance issues as a DBA, you probably will have a plan, I will collect performance counters, I will collect extended events, I will check DMV, I will run these Glenn Berry\u0026rsquo;s scripts to check this and so on. If you\u0026rsquo;re a developer, you will go in different stuff. You will collect traces. You know, DBAs will be still there in 5, 10 years and not a lot of things will change. This is my I believe in this because this is some kind of mine. Yeah, that\u0026rsquo;s some kind of mindset. There is at this type of job which will not be done by other people or by artificial intelligence.\nMS: But you can hear that there is cloud-first cloud-only, all of this stuff with machine learning, with artificial intelligence that you just need to go to the portal, click three times and then everything is done. Do you think that it\u0026rsquo;s possible that it will happen soon like we\u0026rsquo;ve been next? I don\u0026rsquo;t know, five years.\nDR: Yeah, probably. It will happen soon, but DBA will not go away. This is my belief. They will be always a place to work for DBA, Even with all these technologies and artificial intelligence,\nKN: And that\u0026rsquo;s very good news for all the DBA around the world.\nDR: But I\u0026rsquo;m not very good in forecasts. So not so good news.\nKN: So Denis you as a mentor, when you\u0026rsquo;re having a lot of experience, obviously. So what kind of hints would you give to young people who are just starting working on the I.T. Market?\nDR: In the community or on the\u0026hellip;?\nMS: for example, they are just graduating from university and they are just looking for their first work and they would like to work in I.T.\nDR: That\u0026rsquo;s a very good question. You are here for the first work, I would say don\u0026rsquo;t care about anything. Just find the work and go. At this point, when you go out of university, you are looking for a job. At this point, you don\u0026rsquo;t know what you want to do, honestly. So find any job and start work and start doing your best and start to grow in this company. And then you will feel what you like to do, what you don\u0026rsquo;t like to do. And next job, maybe you will fit yourself in this company and everything will be great. If not next job, you will find based on these feelings and this experience, they will ask these questions on the interview. You will learn about this company and it will be better.\nKN: And those are looking at your experience and history on your story, basically, it\u0026rsquo;s maybe at your first or second job would be good to find some mentor or guru or who can learn from. Right.\nDR: Yeah. that\u0026rsquo;s true.\nKN: That can speed up your journey, basically.\nMS: Or maybe even set up the milestone that you need to achieve, because sometimes when you are just after the university, as you have said, probably you don\u0026rsquo;t really know what you really want to do in your life.\nDR: Yeah, that\u0026rsquo;s true. And this is also a very dangerous thing. It\u0026rsquo;s it seems like good and it\u0026rsquo;s good. But you know that when you grow up, your ambitions will be different. Your understanding of the world and your career will be different. And if you said this, go for five 5, 10 years, go in one year, it\u0026rsquo;ll be completely different. It\u0026rsquo;s very good to set goals, but it\u0026rsquo;s also very important to correct the goals from time to time.\nKN: Revisit those goals.\nDR: Yeah, if your goal does not expire, You if you feel that it\u0026rsquo;s not your goal. Yeah. It\u0026rsquo;s not very good for you right now. You should be able to change it every time. And this is very important, especially for young specialists, because you can see like somebody is MVP, I want to be an MVP. This is my go, you go to these. And then you understand: What? This is not what I want. And you should have courage to admit that this is not my goal. I need to set to another. And you can\u0026rsquo;t really understand this until you will go into this path. If you want to be an MVP, go till the end. Try to achieve it. If you feel that this is not your goal at the middle of the path, correct this goal.\nMS: What is the hottest feature that you have on your learning list that you would recommend focusing on? So I would say that are two questions. First one is what is your learning list? And the other one is what do you think that will be beneficial to know in the close future?\nDR: Honestly, I have about 10000 tabs in my favourites bookmarks in my browser. I\u0026rsquo;m not sure if I will finish it till the end of my life with this is my own list. I use Feedly, I check blogs regularly of the top SQL Server specialists and Microsoft blogs. But right now I\u0026rsquo;m doing this probably once per month. I am not very good on this. In this in my top least right now is Postgres internals in my learning at least use reread SQL Server internals book. Yeah, I read it, but not completely. I read a few chapters when they need some information, but they initially from start to end.\nMS: You mean SQL Server 2008/2012 internals by Kalen Delaney.\nDR: Yeah. 2012. I would like to learn something more about Kubernetes and Docker. I feel that this is a very interesting and Linux as well. I am not very good at Linux but they use it from time to time. Well, for SQL Server I have a lot of tabs in my favourites, but I do not have any particular something to learn. And the second question is the cool feature, which I would recommend to learn about.\nKN: Yeah\nDR: Yeah. So if this question, I would recommend not to learn about a particular feature, but I would recommend learning both internals of SQL Server storage engine, SQLOS, Query Optimizer.\nKN: So you mean to learn and know how the engine behind the scenes works.\nDR: Yeah. SQL Server internals is a great book to do with these, to work with this. Also, if it\u0026rsquo;s too hard I would recommend using Ken Henderson books. Those are my favourites. I started from. I still love them. I feel that the information in that book very important and very hard information to understand was given with very simple words so that books are pretty old. Therefore, SQL 2000, but as I remember, maybe I\u0026rsquo;m wrong right now because I became a bit old these days. But and I read this a lot of time ago. But I feel that the information given there is still valid right now.\nKN: Cool. We put all the links, those into these books at the end of the transcript.\nMS: At the end of our conversation, could you please tell us where we can find you? You mentioned that you suspend your activities. But where we can find you in case someone is interested to get in contact with you.\nDR: Facebook probably is the best channel.\nMS: Okay. We will just attach the links. Probably you are also on Twitter, on some other platforms?\nDR: Twitter, Facebook and pretty rare on LinkedIn. I\u0026rsquo;m also going to start my channel, YouTube channel during this commute, self-isolation. I thought about blogging, both probably blogging. I will skip it. Maybe I will blog something.\nKN: So let us know when you started your YouTube channel.\nDR: Videos will be better. Yeah. For me.\nKN: That\u0026rsquo;s cool.\nMS: Thank you very much for joining us today and hope to hear your song.\nDR: Thank you for inviting me.\nKN: Thank you very much, Denis, for sharing your knowledge and sharing your experiences.\nDR: Thank you have a nice day. Bye\nMS: Thanks.\nKN: Thank you, you too.\nUseful links Denis\u0026rsquo; profiles: Twitter | LinkedIn | Blog\n","date":"2020-08-21T07:00:23Z","image":"/uploads/2020/08/asf033-Denis-Reznik.jpg","permalink":"/2020/08/asf-033-denis-reznik-interview/","title":"ASF 033: Denis Reznik interview"},{"content":"Press ADF Release Notes 07/31/2020\nI believe this is the first Release Notes from ADF Team.\nWorking with Git repositories in Azure Pipelines\nMulti-repository access and more.\nDevOps for Azure SQL\nA great cross-section of information on MS SQL database deployment.\nOverview of load-balancing options in Azure\nLoad balancer or Application Gateway or Front Door?\nAsynchronous messaging options in Azure\nCheck differences between Azure Service Bus, Event Grid, and Event Hubs.\nWhat’s new in Azure Cosmos DB’s API for MongoDB?\nFew very useful features announced a couple of months ago.\nWhat are functions in ARM Templates?\nA word from Microsoft’s CEO Video https://www.youtube.com/watch?v=DWoW7ayPzJ4\nAdvancing Synapse - Getting Started with HTAP \u0026amp; Azure Synapse Link\nSmile Corner ","date":"2020-08-02T15:33:21Z","image":"/uploads/2019/11/news-tyler-nix-tPz-uoz5RBU-unsplash-1200.jpg","permalink":"/2020/08/last-week-reading-2020-08-02/","title":"Last Week Reading (2020-08-02)"},{"content":"Press SQL Server Management Studio 18.6 is available\nThe latest version of the most popular SQL Server tools is generally available.\nAzure Data Factory extension\nNew version of Azure DevOps extension for deploying Azure Data Factory.\nBuilding the Ultimate Home Office\nIf you wonder about modernizing your home office, Troy Hunt has a great blog post.\nGarmin is suffering due to ransomware\nIf you think that cloud protect you from everything - think again and check your DR plan.\nMAXDOP in Azure SQL Database?\nNew feature should be available worldwide by August 2020.\nNew hardware options in Azure SQL Database\nNew compute and memory optimized hardware for really demanding workloads.\nNew Business Applications certifications\nIf you are working with Dynamics 365 or Power Platform, consider this certification\nA word from Microsoft\u0026rsquo;s CEO: Video: Benchmarking SQL Server and Azure SQL with WorkloadTools https://www.youtube.com/watch?v=LgVHK9IBXSc\n","date":"2020-07-26T22:58:36Z","image":"/uploads/2019/11/news-tyler-nix-tPz-uoz5RBU-unsplash-1200.jpg","permalink":"/2020/07/last-week-reading-2020-07-27/","title":"Last Week Reading (2020-07-27)"},{"content":"Press Azure resource providers operations\nList of all of the available ADF-specific RBAC permissions.\nSimplifying declarative deployments in Azure\nThe power of What-If within ARM Template deployments.\nCollecting custom parameters/variables for Azure Data Factory deployment\nGreat post from Rayis who is describing in details the process end-to-end.\nWhich database source control model works best for me?\nBenefits and drawbacks of four methodologies.\nSQL Server deployment options for Octopus Deploy\nGreat Alex\u0026rsquo;s comparison of all deployment options for Octopus.\nMAXDOP is a Lie\nAre you sure you are aware of how this option works in SQL Server?\nBenchmarking Azure Synapse Analytics – SQL Serverless, using .NET Interactive\nSingle File or Multiple? Parquet vs CSV? Parallel or Series?\nEncrypt Azure Data Factory with customer-managed keys\nSecurity feature.\nMoving A Database to New Storage With No Downtime\nVery smart advice from smart Bob. Is it even possible? Check this out.\nI want to skill up – but what do I learn?\nFew Microsoft MVPs share their thoughts on what topics you should consider learning about.\nADF Adds Connectors for Delta Lake and Excel\nUse ADF to natively read from Excel spreadsheets and use the data flow Delta connector (public preview) to create Delta Lakes, manage data transformations, updates, upserts, deletes using Delta.\nA word from Microsoft’s CEO Video Azure Data Factory | Publish from code with one task in DevOps Pipeline (2/2) Smile Corner ","date":"2020-07-12T21:34:26Z","image":"/uploads/2019/11/news-tyler-nix-tPz-uoz5RBU-unsplash-1200.jpg","permalink":"/2020/07/last-week-reading-2020-07-12/","title":"Last Week Reading (2020-07-12)"},{"content":"Struggling with #ADF deployment? adf_publish branch doesn\u0026rsquo;t suit your purposes? Don\u0026rsquo;t have skills with PowerShell? I have good news for you. There is a new tool in the market. It\u0026rsquo;s a task for Azure DevOps Release Pipeline to deploy whole ADF from code (JSON files) to ADF instance in Azure. Behind the scenes, it runs the PowerShell module which does all job for you.\nSounds unbelievable? But it\u0026rsquo;s real! Check it out for yourself.\nThis video is the second part (out of two) and explains how to extend the deployment scenario by adding further environment and what to do to replace all required properties of ADF objects to get an isolated environment as a result.\nThe first part explained how to create Build Pipeline and Release Pipeline with this task. How to configure the fields and check how it works.\nUseful Links DevOps extension in Visual Studio marketplace: Data Factory Tools\nScreenshot ","date":"2020-07-07T07:29:55Z","image":"/uploads/2020/07/ADFTools2_cover_1200px.jpg","permalink":"/2020/07/publish-adf-from-code-to-further-environments/","title":"Publish ADF from code to further environments"},{"content":"Press ASF 032: Jennifer Stirrup interview\nThe latest episode of the Ask SQL Family podcast.\nAzure SQL Managed Instance MP\nNew Management Pack for System Center Operations Manager that allows you to monitor Azure SQL Managed Instance.\nSQL Server CU #21 for SQL Server 2017\nCumulative Update #21 for SQL Server 2017 that fixes several issues with SQL Engine and AlwaysOn High Availability.\nSQL Server CU #5 for SQL Server 2019\nCumulative Update #5 for SQL Server 2019 that fixes several issues with Big Data Clusters and Polybase.\nFree Azure replica for SQL Server with DH2i\nIf you have Software Assurance consider using free SQL Server replica in Azure.\nHow MAXDOP Really Works\nHave you wondered how MAXDOP works under the hood? This great article explains it in details.\nFree course on Pluralsight about Azure Data Explorer\nLearn how to ingest, query, analyze and visualize large amounts of data with Azure Data Explorer.\nMicrosoft Teams June 2020 update\nNew features including 7x7 gallery view, breakout rooms, multi-window for meetings and calls and much more\nA word from Microsoft\u0026rsquo;s CEO: Video: The Case of the Unexplained https://www.youtube.com/playlist?list=PLhFhDWFYccZ9eb0ND71IZyLCB4IRL21R2\n","date":"2020-07-05T21:00:43Z","image":"/uploads/2019/09/news-alex-Yvl081TVnvA-unsplash-1200.jpg","permalink":"/2020/07/last-week-reading-2020-07-05/","title":"Last Week Reading (2020-07-05)"},{"content":" Introduction Jennifer Stirrup is a data strategist and technologist, Microsoft Data Platform Most Valuable Professional (MVP), Microsoft Regional Director, Microsoft Certified Trainer, founder of Data Relish Ltd, diversity and technical community advocate, public speaker and blogger. She is a published author, writing two books on Tableau and R, and she is in the process of writing another on Microsoft Power BI.\nKeynote speaker, author, diversity, influencer.\nDue to the coronavirus outbreak, this talk has taken place online on 26 June 2020 (Friday) using Zencastr platform.\nInterviewer: Kamil Nowinski (T).\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript Unfortunately, this time working on the transcript requires more time and effort, because of this it has not yet been completed.\nI hope to publish it later this month.\nFeel free to reach me out if you would like to help me complete this out.\nApologize for the inconvenience.\nUpdate I\u0026rsquo;m really sorry that we lost a very valuable Microsoft MVP member. Jen resigned from her MVP title a few days before the renewal process on 1 July 2020:\nI have stepped down as a Microsoft MVP\nUseful links Jen\u0026rsquo;s profiles: Twitter | LinkedIn | Blog\nA community of pro bono data experts: datakind.org.uk\nThe first computer of Jennifer: ZX81\nJennifer is an author of the following books:\nTableau: Creating Interactive Data Visualizations\nTableau Dashboard Cookbook\nAdvanced Analytics with R and Tableau\nRecommended books:\nGo Tell It on the Mountain by James Baldwin\nInvisible Man by Ralph Ellison\nThe Bone Shard Daughter by Andrea Stewart\n","date":"2020-07-02T23:55:24Z","image":"/uploads/2020/07/asf032-Jennifer-Stirrup.jpg","permalink":"/2020/07/asf-032-jennifer-stirrup-interview/","title":"ASF 032: Jennifer Stirrup interview"},{"content":"Press ExpandString in Powershell\nThis function can do much more than you think.\nPester - using -should -throw\nUseful example when you want to filter output by ExceptionType or ErrorId.\n[ADF: Include Headers in Zero Row Data Flows [UPDATED]](https://kromerbigdata.com/2020/06/14/adf-include-headers-in-zero-row-data-flows/)\nIt has been a bug, but now you can learn to build mapping data flow in ADF.\nMapping Data Flows in Azure Data Factory\nThis project use case uses Covid-19 publicly available datasets and ADF MDF and compares it with Databricks and SSIS.\nApache Spark Connector for SQL Server and Azure SQL is now open source\n\u0026hellip; and the code is on GitHub available, obviously.\nUsing Python and Monte Carlo to predict my Student Loan Repayment\nA Data Science approach to managing personal finances.\nRecommended naming and tagging conventions\nDo your names stick to the MS recommendations?\nA word from Microsoft’s CEO Video: https://youtu.be/2vcXIiSD8-Y\nAzure Data Factory | Publish from code with one task in DevOps Pipeline (1/2) Smile Corner ","date":"2020-06-28T22:22:41Z","image":"/uploads/2019/11/news-tyler-nix-tPz-uoz5RBU-unsplash-1200.jpg","permalink":"/2020/06/last-week-reading-2020-06-28/","title":"Last Week Reading (2020-06-28)"},{"content":"Struggling with #ADF deployment? adf_publish branch doesn\u0026rsquo;t suit your purposes? Don\u0026rsquo;t have skills with PowerShell? I have good news for you. There is a new tool in the market. It\u0026rsquo;s a task for Azure DevOps Release Pipeline to deploy whole ADF from code (JSON files) to ADF instance in Azure. Behind the scenes, it runs the PowerShell module which does all job for you.\nSounds unbelievable? But it\u0026rsquo;s real! Check it out for yourself.\nThis video is the first part (out of two) and explains how to create Build Pipeline and Release Pipeline with this task. How to configure the fields and check how it works.\nhttps://youtu.be/2vcXIiSD8-Y\nThe second video will extend this scenario by adding further environment and explains what to do to replace all required properties of ADF objects to get an isolated environment as a result.\nUseful Links DevOps extension in Visual Studio marketplace: Data Factory Tools\nScreenshot ","date":"2020-06-26T08:10:00Z","image":"/uploads/2020/06/ADFTools1_cover.jpg","permalink":"/2020/06/publish-adf-from-code-to-service-easily/","title":"Publish ADF from code to service easily"},{"content":"Press New AI-900 exam\nIf you don\u0026rsquo;t feel comfortable with AI check online learning paths available in the middle of the page.\nNew DP-900 exam\nWant to check yourself for new Azure Data certification? Take new Microsoft Azure Data Fundamentals exam.\nAzure Data Studio - June 2020 release\nNew version of the well-known multi-database, cross-platform desktop environment for data professionals.\nAzure Data Studio Extension\nSQL Server Assessment Extension for Azure Data Studio provides a user interface for evaluating your SQL Server instances and databases for best practices.\nAzure responds to COVID-19\nAwesome video from Mark Russinovich how Microsoft Azure responded to COVID-19.\nSQL Server on Virtual Machine\nWondering how to setup VM for SQL Server? Check this book for best practices.\nA word from former Microsoft\u0026rsquo;s CEO Video https://www.youtube.com/playlist?list=PLr9ab4Dj3ObsGA8jdstJFZfidQ51DvPvu\n","date":"2020-06-20T10:13:24Z","image":"/uploads/2019/11/news-tyler-nix-tPz-uoz5RBU-unsplash-1200.jpg","permalink":"/2020/06/last-week-reading-2020-06-21/","title":"Last Week Reading (2020-06-21)"},{"content":"Automation is a useful thing.\nIt allows to avoid many mistakes and saves a lot of time. However, you need to invest some time to get started.\nAt least the basic knowledge of PowerShell, CLI, Azure DevOps and security issues is useful.\nBut what to do when these skills are missing in the team? How to start? Where to start?\nIn this video, I will show you briefly how to automate the deployment of service in Azure.\nLinks In my local machine, on top of Git, I use TortoiseGit in order to manage code with Git repository:\nTortoiseGit: https://tortoisegit.org/\n","date":"2020-05-28T12:44:37Z","image":"/uploads/2020/05/arm-templates-video-q.jpg","permalink":"/2020/05/automate-deployment-of-azure-services-with-arm-template/","title":"Automate deployment of Azure services with ARM Template (video)"},{"content":" Introduction Paul Andrew. Principal consultant and architect at Altius specialising in big data solutions on the Microsoft Azure cloud platform.\nData engineering competencies include Azure Data Factory, Data Lake, Databricks, Stream Analytics, Event Hub, IoT Hub, Functions, Automation, Logic Apps and of course the complete SQL Server business intelligence stack.\nMany years’ experience working within healthcare, retail and gaming verticals delivering analytics using industry-leading methods and technical design patterns.\nSTEM ambassador and very active member of the Data Platform Community delivering training and technical sessions at conferences both nationally and internationally.\nFather, husband, swimmer, cyclist, runner, blood donor, geek, Lego and Star Wars fan!\nThis talk has taken place in London, UK on 9 March 2020 (Monday).\nInterviewer: Kamil Nowinski (T).\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript KN: Today is a special guest with me, very well experienced data engineer and architect and also my company mate now, which is…\nPA: Paul Andrew. Hello everybody.\nKN: Paul, could you introduce yourself from the name but also where do you live?\nPA: My name is Paul Andrew. I\u0026rsquo;m a married, non-smoker husband, two children. The personal things but, I currently live just North of Birmingham in the Midlands. A lot of people from London so I\u0026rsquo;m a northerner but no actually I live in the Midlands. There is many other places further north than I am.\nKN: Come on, only the Britons knows those parts of UK.\nPA: In terms of professional wise. Yes, as Kamil says principal consultant at the moment since solution architects working with Altius. Probably about 10 to 15 years of experience now in the industry. Starting very much as on-premises SQL Server BI Developer, maybe originally, and then there\u0026rsquo;s the technology is involved, my role has evolved and coming into various consultants is now and things are obviously progressed and got a lot more crazy since then since we started working with Azure and what are job titles are and what we do.\nKN: Exactly. What actually do you do during your business day?\nPA: Interesting one. At the moment I\u0026rsquo;m the solution lead for delivery team, for Altius\u0026rsquo;s retail customers. So I will support the team with the design of the solution, getting the next bits of work lined up. You know, attending the Sprint Ceremony is to do the refinement of the store is doing a lot of the design documentation working with the customer requirements. So, really. As far as supporting customers goes, that\u0026rsquo;s solution lead role is where I\u0026rsquo;m at the moment doing the architecture, supporting that delivery team. Which at the moment that delivery team is split between India between the Netherlands, partly in the UK in London and then the customer is based in Belgium. So, it\u0026rsquo;s quite an interesting set up at the moment with the remote teams.\nKN: So mainly working from the UK? How often do you fly to the Netherlands?\nPA: Sure, I\u0026rsquo;ll predominantly based at home, so working from home a lot of the time during a two-week sprint and then probably go over to Belgium once a week. Maybe to see the team in Amsterdam as well probably every fortnight so. That\u0026rsquo;s the current set-up, but sometimes yeah, it does feel like I am a visiting Birmingham Airport far too much.\nKN: But working from home most of the time. Lucky Man.\nPA: Yes. I mean, my youngest daughter, she\u0026rsquo;s just turned one so having her get back from nursery come into the office and start pulling on the chair and bashing the keyboard. And so it was a nice way to be told: “It\u0026rsquo;s time to finish work”.\nKN: I still remember when you were speaking at SQLBits with Chris Testa O’Neil. It was most probably not your first time when you were speaking but that time or around the time you\u0026rsquo;ve been awarded with MVP title. It was probably a few months after.\nPA:. I think I was awarded my MVP in sort of June/July time of 2017, I think. It was just before they actually changed the renewal cadence. I think, I just kind of got in there before. They then started the new March sort of renewal cycle. But, it was very much a Microsoft session that Chris was leading around Data Factory. I think you know just talking about the I think it was probably the V1 version of Data Factory…\nKN: Yeah, yes, definitely, it was V1. I remember the complex diagram that you were showing.\nPA: So, I mean I talked about Data Factory in terms of the good, the bad and the ugly at the time because it was version one of the products and Chris introduced the tool and the resource. And then I just started, kind of I think 10 minutes at the end from just real-world experience having implemented it.\nKN: The speaking what does commitment in the community mean for you?\nPA: Lot\u0026rsquo;s of things I guess. I can safely say that I\u0026rsquo;m definitely passionate about what I do, and I love sharing that knowledge. Sharing that passion with other people. So for me, the community is the best way to do that. I really do love speaking. I get nervous before every single talk.\nKN: Like we all, I think. Which is probably good.\nPA: I think it\u0026rsquo;s good. I think that a little bit of nervousness keeps you sharp. It doesn\u0026rsquo;t matter how many times you spoken yet. I still get nervous. But for me the community is that way to share that passion. To share that knowledge. And that\u0026rsquo;s why I like doing.\nKN: Good to know that it\u0026rsquo;s not only me, being nervous before speaking. Only the native English-speakers as the same thing.\nPA: It could be an audience of 10 people, it could be an audience of 400; for me, the nervousness is the same as the time.\nKN: I know that you record and report all your activities in what kind of form. Could you tell us a bit more about that? Who is maybe not familiar with or not having an account on Twitter or something?\nPA: Sure, so when it comes to the MVP recognition is that what you mean?\nKN: It\u0026rsquo;s a bit related, but I\u0026rsquo;m talking about your Power BI reports.\nPA: Sure. I guess really for my own sanity, I started keeping a log of where I had spoken around the UK at the time, and then as I attended more events around the world. So I think I kept this log this database and then playing around I put the Power BI dashboard on the front of that and I think, given where right now we\u0026rsquo;re saying that this is my 100th talk.\nKN: 100th exactly? This one?\nPA: Yeah. Last week I was over at SQL Conference in Germany and then SQL Saturday Belgium. So I think if my database is correct then this will be my 100th community talk in the last six or three or four years so.\nKN: Amazing, I\u0026rsquo;m so now very proud.\nPA: Thank you, thank you for having me.\nKN: Fantastic, 100, good. And yeah, if some people would like to or interested in see your report they can check. That\u0026rsquo;s probably all. Find it out on some on your Twitter. Probably in some timeline.\nPA: There\u0026rsquo;s a code GitHub repo for it. I think I just called community speaking log and the database scripts and the latest version the Power BI dashboard is in there, if you want to see where I\u0026rsquo;ve been.\nKN: I put the link to that as well.\nKN: So, MVP since 2017, so we both know that there\u0026rsquo;s a big sacrifice behind the being an MVP member. But also there\u0026rsquo;s are some benefits to it. How did you use it? What does it mean for you?\nPA: Good question. So I think I\u0026rsquo;ve said this to Microsoft as well, but I mean there all the benefits that you get from being MVP. The free clothes, the Azure credits, all those things, for me I would trade all of those benefits for the interaction you get with the Microsoft product team. They listen to you about what changes should be coming in the tack, and they also share what they\u0026rsquo;re doing with it. So, that\u0026rsquo;s the NDA, that non-disclosure agreement that you have with Microsoft for me that is the biggest benefits of being an MVP. More so than the free clothes and things I joke about…\nKN: The free clothes are actually not free. I mean you need to pay for a delivery, which is in most cases much more expensive than the clothes itself.\nPA: Yeah, if you\u0026rsquo;re not in the US, you have to pay to get a jacket.\nKN: Which is funny enough.\nPA: Did I answer your question?\nKN: Ahh, yes, thank you.\nKN: So under NDA you have the information, the updates from the first hand.\nPA: So it\u0026rsquo;s good to know what\u0026rsquo;s Microsoft working on, what\u0026rsquo;s coming next, and you know having that insight, it\u0026rsquo;s definitely helped me shape my career a little bit in some cases and then help my customers in terms of what we deliver for them. It\u0026rsquo;s a really nice relationship that you get with Microsoft, which gives you that future look.\nKN: Even having art of updates from MVP members, I mean we as a community and also I\u0026rsquo;m being and feeling part of this data platform things. I am talking about that we couldn\u0026rsquo;t avoid, you know, having a Data Factory in version 1, now, we have version two. Somethings happens anyway.\nKN: MVP Global Summit supposed to happen next week, but due to the corona-virus it\u0026rsquo;s will be held online only. Do you regret?\nPA: Sure, I think it\u0026rsquo;s a great event. It\u0026rsquo;s a great networking opportunity for everybody that goes to Redmond for the MVP Summit. And you know, as I\u0026rsquo;ve already suggested, that that interaction you get with the product teams, that\u0026rsquo;s definitely something that\u0026rsquo;s better face to face. But, I mean we are a nomadic people in the IT industry, so it being an online event it\u0026rsquo;s not great, but I guess here with my young family there are some benefits to not spending nearly 24 hours in an airport to go there and back. But, I will miss the MVP Summit being in Redmond this year.\nKN: And place, and other people and a need to experience talking and etc.\nPA: Yeah, the dates for the 2021 once Summit have just been announced, so looking forward to allow to spin the next chance to go to the Microsoft HQ.\nKN: But it\u0026rsquo;s one year… we need to wait.\nKN: I love the numbers and accidentally we have changed our job pretty much in the same time. Like about six months ago.\nPA: Yeah, you did copy me?\nKN: Ha ha, no, I haven\u0026rsquo;t known that time that you are changing your job. And after being working 18 months in the previous company, we both. we\u0026rsquo;re working in 18 months.\nPA: Yeah, I changed a couple of times. With Purple Frog, I think two years and then with Adatis, I think about a year and I changed to Altius.\nKN: So we both changed to Altius company right now, another consultancy company. You have more experience than me in consultancy company. What do you think about that kind of changes? Do you see any differences working for just different company in consultancy market for customers?\nPA: Absolutely. Something that I was maybe a little naïve to once upon a time was how calculating, I think, Microsoft or when it comes to aligning customer requirements with one of their partners and certainly having been with a few different consultants is now of different sizes just how calculating that relationship is and being with Altius now engaging with some of the larger enterprise customers, it\u0026rsquo;s really interesting to see some of the challenges that they have. None of which most of the time, our technical challenges. More of which are challenges around policy and bureaucracy and all those other sorts of things. And you know, the team I\u0026rsquo;m currently working with now we are all remote, so it\u0026rsquo;s those new sorts of challenges, which is what I\u0026rsquo;ve certainly found in this latest move.\nKN: That\u0026rsquo;s interesting. So you mentioned that you were working remotely most of the time because you are neither not living in London and nor around. So how much it takes you to come to London?\nPA: How much does it take to come here? Not too long actually, so there\u0026rsquo;s for me it\u0026rsquo;s quite nice so I get a direct train from Stafford that comes straight into Houston and that only takes an hour and 15 minutes…\nKN: From door to door?\nPA: From door to door, probably 2 hours. So maybe you know a 30-minute drive home to the train station, then maybe 20 minutes on the Northern line and Circle Line ground to Blackfriars. So, probably 2 hours door-to-door, which I think given how far North I am in the country. I know I have family just outside the M25 in St. Albans and for them to commute into London can take an hour and a half on it…\nKN: compared to where you living is like almost nothing but it\u0026rsquo;s still 2 hours in one direction here.\nPA: So I think, it\u0026rsquo;s relative. I don\u0026rsquo;t mind it. So I come into the office probably every week.\nKN: How about your work-life balance then? How you can cope with that?\nPA: I wouldn\u0026rsquo;t call it a “balance”. It\u0026rsquo;s a hard one. I mean, I have the advantage of we\u0026rsquo;ve talked about of working from home, which is nice a lot of the time and I think if anything that probably makes up for some of the travellings that we do for customers for community events or just come into the office. But I think this is the life of any consultant really. There\u0026rsquo;s two parts to it I guess: there\u0026rsquo;s a work-life balance when it comes to travelling and then there\u0026rsquo;s also a balance when it maybe comes to some of the community commitments that we have around blogging and speaking and all these other things. And it\u0026rsquo;s hard. It\u0026rsquo;s a sacrifice but safe for me it\u0026rsquo;s that passion, that kind of fuels that and the rewards that we\u0026rsquo;ve spoken about the MVP program is what kind of makes some of that worthwhile. It\u0026rsquo;s a trade-off, I guess. It\u0026rsquo;s hard to call it a balance, I don\u0026rsquo;t think there\u0026rsquo;s a balance.\nKN: Yeah, because you mentioned travelling, commitment to the conferences. Where is the time for life or private life for the family?\nPA: I think if I had a hobby I might struggle, but I think I realized probably quite early on and probably this is where the passion comes from, that to some extent my job is my hobby.\nKN: That’s my next question: how do you know? So what\u0026rsquo;s your hobby data?\nPA: I mean well… wouldn\u0026rsquo;t so upon a time, I\u0026rsquo;m I used to swim a lot, I used to running half marathons. I used to really enjoy biking, I used to be on the darts team, but all of these things have kind of slipped away from here in the pursuit of a career maybe and also because of a young family takes that time so\u0026hellip;\nKN: The other things disappeared.\nPA: Yeah, yeah, I don\u0026rsquo;t know if I have any hobbies anymore. Maybe, just… sleeping is my hobby…\nKN: I like this also, but there\u0026rsquo;s not too many time to having that hobby, to practice that hobby.\nPA: Absolutely.\nKN: So conferences and event is also great opportunity to visit new places. So how often do you take your family to conferences like SQL Saturday, like past weekend for example?\nPA: As often as I can, so these events they are a little bit for me in the networking and you know for the sharing but also trying to make them miniature Holidays for the family. My wife Christine, She\u0026rsquo;s been to quite a few events now. She\u0026rsquo;s been with me to SQL Saturday Iceland. That was the main one because she really enjoyed that. She\u0026rsquo;s a geography teacher, so Iceland is a great place to visit for lots of reasons. But I mean our daughter was born in December of last year, so sorry, the year before, she\u0026rsquo;s 15 months old now. So yeah, we\u0026rsquo;ve not been to as many because we\u0026rsquo;ve had a baby. But I mean, in theory, our daughter, she came to SQLBits, when it was the Speakeasy theme, I’m losing track of what year that was, whether that was last year, or the year before. But the family always gets it comes to community events if logistics and things work out, \u0026lsquo;cause I like to bring them along.\nKN: So in that way also you can spend, you know, a little bit time and weekend with your family in other places, interesting places.\nPA: Yeah, it\u0026rsquo;s definitely a life goal of mine to try and visit every single country in the world, but that\u0026rsquo;s obviously very ambitious, but I\u0026rsquo;d like to. And if the family can come as well then great.\nKN: OK, how are you preparing yourself for a speech?\nPA: How do I prepare? Lots of ways. I will always go through my slides multiple times. For me, I don\u0026rsquo;t know if this is just me or not, but when I build my slides and the way I build them in quite a visual way. I know the act of creating the slides for me\u0026hellip; it helps me think about what I\u0026rsquo;m going to say. So that\u0026rsquo;s kind of one thing, just building the slides for me teaches me what I\u0026rsquo;m going to talk about. But then I will go through the slides multiple times and practice, practice, practice because you have to. I will also run through them, I will present them to the wife, to the family sometimes. I will record myself and listen to myself in the car or on the train, which I hate the sound of my own voice, so this I will be in the car and I\u0026rsquo;ll probably be cringing and somebody next to me will probably look at me. But listening to myself sometimes helps. There\u0026rsquo;s lots of techniques. Depends on the talk.\nKN: If you\u0026rsquo;re working from home you don\u0026rsquo;t have too many opportunities to travel by car. So, when do you do that?\nPA: And sometimes small train or plane or just recording myself going through the talk is one way of doing it. Rehearse in the demos and doing screen recordings of the demos and things and then watching them back is a good one.\nKN: So are you recording only the audio, or also the video? How the body works and how you are presenting with your body? Or just in that part of preparing it doesn\u0026rsquo;t matter?\nPA: I think just the audio really, when comes to actually being on stage and kind of the problems of the slides. It\u0026rsquo;s maybe a little different. There\u0026rsquo;s no preparation there, that\u0026rsquo;s just all adrenaline.\nKN: All those are at least maybe since I remember, probably since Data Rally, it was called at that time SQL Relay, like two years ago, you started your icon game at the beginning of each session. So what\u0026rsquo;s your favourite service in Azure then?\nPA: My favourite service. I don\u0026rsquo;t know if I have a favourite. I mean, let\u0026rsquo;s go the easy one then. Let\u0026rsquo;s say Data Factory is now.\nKN: Exactly. I know that you\u0026rsquo;re speaking and speaking, maybe not a lot, but, many times at least, and also recently writing a blog posts about it many times, so started also the serious about ADF, etc. And I remember that you started working with version one, so you have experience with that, which I haven\u0026rsquo;t. And that service would be obfuscated by Microsoft right now, yes?\nPA: Yeah, I don’t think you can create a new Data Factory V1 instances anymore or in certain if you have a V1 instance, I think you need special permissions now to do these sort of authoring by the portal. So but you\u0026rsquo;re right, I started working with Data Factory, I think in 2016 or sort of late 2015 when V1 was still in preview. So, it\u0026rsquo;s been a long time with Data Factory.\nKN: So now you have experience with version 2. How you could compare both of those versions?\nPA: Comparing V1 and V2? I mean they\u0026rsquo;re very different services. The goal of course being the same, but I don\u0026rsquo;t know. There is things about version one, I think that I probably still miss. There are things that in version two that, of course, a lot better. The number of activities we can now use, the dynamic content that we can build into pipelines is great, and the UI, I think in version two makes things really nice for a developer to put things together.\nKN: But there\u0026rsquo;s still lack of UI, like in terms of having UI in Visual Studio, for example, not in the browser that many of users complain about it.\nPA: Yeah, having that Data Factory projects that you got in V1 in Visual Studio 2015 is something that I do still miss, because ultimately it means that you can\u0026rsquo;t work on Data Factory when you are on a plane or a train and travelling quite a lot as I do. It means I do need a good, reasonable Internet connection to actually work in the browser to develop the tool. So, I mean that\u0026rsquo;s for me the biggest thing I would miss about V1 and you know, there\u0026rsquo;s a new answers around the deployment process, which I think were slightly better when you publish the solution in V1 of Data Factory versus how you do the ARM Template deployments now in V2. But yes, it\u0026rsquo;s a good tool. It\u0026rsquo;s just these differences.\nKN: It’s a different way how to use it, yeah?\nPA: Yeah.\nKN: I think it\u0026rsquo;s worth mentioning that the ADF V2 is very well integrated with Git repository in general. So it\u0026rsquo;s also helps you to manage the code, the first thing. But also it helps you somehow do the deployment through the “ADF_publish” branch which basically is created when and you publish things. But don\u0026rsquo;t you think that it\u0026rsquo;s overcomplicated a little bit?\nPA: I don\u0026rsquo;t know about overly complicated, but I think certainly for somebody that is used to interacting with source code repository via Visual Studio where they have that you know that offline copy that local version of the code and they do the changes that commits the pushes and things. I think if you come from that environment to then working with the online tool is quite interesting and probably different because you know you can be working in a feature branch in Data Factory and every time that user interface changes or becomes dirty, you click save behind the scenes it is doing a commit every time to the GIT, so it can make the version history of a particular file looking very chatty, when somebody is looking at the source code repo. So, I\u0026rsquo;m not sure about complicated, but certainly different to probably have other tools and how other programmers might interact now with source control.\nKN: Have you had that experience, you know, when you have quite so well complicated or complex, branching strategy, having more than one or even more than two (branches), like “master” and “developer” and at the end, there\u0026rsquo;s only one “adf_publish” branch.\nPA: Yeah. So that I mean I we used to get flow branching strategy now as probably a concept. But, Data Factory doesn\u0026rsquo;t really align to that. So I think the common way we get round it is by not using the content from the “adf_publish” branch, but actually using the component parts of Data Factory in the JSON files.\nKN: So, actually the codes which is saved and holds in the branch. So iteratively going through all the objects and deploying them one by one.\nPA: Yeah, so like very much what Data Factory V1 did with Visual Studio 2015 project when you hit publish. You select as what parts of the Data Factory wants to deploy and it did it. We kind of take a very similar approach now with V2 to target, which bits of the JSON we want to deploy.\nKN: OK, so in general - do you have feelings which version is better one to deploy things?\nPA: That\u0026rsquo;s why it “depends” questions. I think it\u0026rsquo;s got to come down to the complexity of your environment. You know, if you have dedicated: DEV, TEST, PRODUCTION, UAT, PRE-PRODUCTION, however many environments you have and I think that complexity, but then also the complexity of your Data Factory. Do you have 100 pipelines or do you have three? I think if you\u0026rsquo;ve got two environments and three pipelines, then maybe just using the publish option and arm templates is easier. If you have 16 environments and 100 pipelines and many, many activities then yeah, probably using the underlying JSON files might be a better option for you.\nKN: Azure SQL Data Warehouse has been renamed to the Azure Synapse.\nPA: Yes.\nKN: OK, and do you have access to that?\nPA: No.\n[We both have access to private preview version of Azure Synapse in shared pool for MVPs]\nKN: Yes, me neither. Also, SQL Server 2019 Big Data Cluster - how do you think: what\u0026rsquo;s opportunity those technologies gives to the customers these days?\nPA: Good question. I think, or my perception in both cases, it\u0026rsquo;s where Microsoft is now moving forward and they\u0026rsquo;re fusing their tech with the open-source world in a lot of cases. Bob Ward got a great podcast they did for the Azure Podcast around Big Data Clusters and he talks about all the good bits of those tools coming together and what they\u0026rsquo;ve kind of put into that new product. So, I think for me, the Big Data Cluster offering and what we hope is the Synapse offering, it is that a fusion of Microsoft SQL bits with the open-source good bits and bringing together a suite of tools that could probably deliver 90% of what the customer needs. It\u0026rsquo;s interesting times to see how those two things will affect the data engineering landscape or certainly my job.\nKN: So yeah, working with Azure right now, might be quite challenging to back to on-prem when some customers have more servers on-prem and, you know, to install SQL Server 2019, but yeah, maybe it also quite a challenge and goods benefit of that.\nPA: I mean if they have hardware than I can just deploy Kubernetes and they can use it on-premises until that hardware becomes obsolete and then they could always migrate that Kubernetes instance elsewhere if they need to.\nKN: Oh yeah, definitely. A much easier way to migrate the things to the cloud.\nPA: Sure.\nKN: What do you think, what is the biggest challenge in IT those days?\nPA: The biggest challenge in IT? That\u0026rsquo;s of a large question. I don\u0026rsquo;t know if I have time to answer such a big question.\nKN: Or maybe, you know, the biggest challenge in the clouds in general or specifically in Azure.\nPA: So, maybe I\u0026rsquo;ll draw on the talk I gave at SQL Saturday Belgium, so maybe help answer that. My talk was about an architect sky to delivering data insight. So I think to answer the question what is the biggest challenge… I guess for most people it\u0026rsquo;s what service do I use? What tool is right for the situation that I\u0026rsquo;m working with now, because there\u0026rsquo;s so many different ways you can build and deliver things now within the within Azure and all the capabilities that you have there.\nKN: You can deliver the same software, the same service with using different services.\nPA: Yeah, it\u0026rsquo;s really about knowing what not to use, when and what is the right tool. I always say to my customers that don\u0026rsquo;t be distracted by the new and shiny technology and make your requirements fit the technology. It\u0026rsquo;s about the other way around. It\u0026rsquo;s about taking your requirements and making sure that you choose the tech that fits what you need to do rather than you have been distracted, maybe with new things.\nKN: I\u0026rsquo;ve been on your session two days ago and you were talking also about constantly changing requirements. So how to be prepared for that?\nPA: It\u0026rsquo;s a good one. So this is a fault of customers, I think this is just always the case that you deliver one thing and then it means that you end up delivering more and more because requirements change. But I guess it\u0026rsquo;s about trying to have a more of a plug and play architecture having that flexibility to bolt on those new requirements and not being too rigid with you delivery approach, which means you can add more two things as you need. I don\u0026rsquo;t think this is a one good methodology that will allow that, but certainly some of the tools that you got in Azure now they do give you that flexibility just to add more and support those changing requirements, but it\u0026rsquo;s a challenge either way.\nKN: Yes, it\u0026rsquo;s always a challenge. But do you think it\u0026rsquo;s maybe is a way to prepare for that or maybe avoid that always changing requirement? Maybe, if the first requirement comes, maybe we can do something better to avoid the second iteration of set of ideas.\nPA: Sure, I mean, maybe a crystal ball is the thing that we do better. Somebody that can look into the future and say “we know every single requirement that\u0026rsquo;s coming”. In some cases, I think, the now you can plan ahead, but it\u0026rsquo;s I don\u0026rsquo;t have a good answer, really, I think. In some cases, changing requirements the technology allows it, it\u0026rsquo;s very easy. You know the case is changing requirements means that you almost need to start again. It would be great to be able to predict all of those things. I don\u0026rsquo;t know if we can.\nKN: What do you think what is the best way to be up-to-date with technology?\nPA: The best way to be up-to-date with technology… the technology we use or just any technology?\nKN: Yes, we use and we are working with.\nPA: I would say just immerse yourself in. It\u0026rsquo;s just every day. Try and try and keep up with all the latest news and announcements where you can. I listen to quite a lot of podcasts in the car to try and keep up with things. But just I think almost you could say, except that you\u0026rsquo;re never going to know everything, just keep trying and just keep running along to try and keep up with it. But I feel like in the IT industry now we\u0026rsquo;re going to end up in a situation. Probably a lot like professional footballers, where you know they retire at the age of 35, because they\u0026rsquo;re not young people anymore in their industry and they can\u0026rsquo;t keep up. I think probably will have the same problem in the IT industry that it\u0026rsquo;s only young people that they can keep up with the pace of change.\nKN: That’s right, but with a small difference. We will not be as rich as much as those people. [laugh]\nPA: No, we definitely won\u0026rsquo;t be worth as many as footballers, no. But I think I can see by the time I get there maybe 40, 50, I won\u0026rsquo;t be able to keep up with the IT industry anymore and I\u0026rsquo;ll have to rethink my life.\nKN: That\u0026rsquo;s true.\nKN: The last question: what hints would you give to young people who wanted to start working on the market, on the IT market specifically. At the beginning of their journey?\nPA: I would say: don\u0026rsquo;t be afraid to get it wrong. Go and figure out what you like doing, what you don\u0026rsquo;t like doing, and start eliminating your options because the IT industry now is so broad in terms of the job roles that you have. Would not so upon a time you could have a system admin that looked after all parts of an IT network and now the breath of the industry it\u0026rsquo;s so massive. And then the depth in which you can specialize in so many different areas. So I think it\u0026rsquo;s a case of going make some wrong decisions, figure those things out, and then find out what is the right decision. A little bit of trial and error almost and say not being afraid of that trial and error.\nKN: Which might be tricky for young people, especially if you\u0026rsquo;re starting working.\nPA: I mean you know somebody that\u0026rsquo;s just completed a University degree in computer science, and you know that breath of that and the scope what you do next, it\u0026rsquo;s a hard one. I started out in the village where I was living, repairing friends and family’s computers and setting up printers and putting RAM in computer towers and things.\nKN: I can see that we all have such history…\nPA: Yeah, unfortunately, I\u0026rsquo;m still go-to guy within the family when it comes to Technical Support of all things from “my Word document isn\u0026rsquo;t landscape or portrait” to “my website is broken and my printer needs setting”. So, I\u0026rsquo;m still that guy, unfortunately. But I think that for me helps me figure out what parts of the IT industry I wanted to then focus on.\nKN: And at the end of our conversation, could you tell us where we can find you. Were people can find you?\nPA: Where the people can find me?\nKN: I\u0026rsquo;m not talking about your private address. (laugh)\nPA: No, haha… this is “my dress”… My blog is MrPaulAndrew.com and then from there yeah you could probably get my email and my various profiles and things. But my blog is probably the best landing point and reach out to me.\nKN: Cool. Thank you. But, do not your blog ends with “.tech”?\nPA: My blog ends with all things. I say that I play Internet wars with another Paul Andrew that\u0026rsquo;s out there. A chaper, I think is far more famous to me actually, but I think he designed shoes. So this shoe designer has something “paulandrew.com” and then what I\u0026rsquo;ve tried to do is I\u0026rsquo;ve tried to buy up all the other “PaulAndrew” domains and they all do I think redirect to my blog. mrpaulandrew.com was the 1st and I\u0026rsquo;ve registered a few more just to play Internet wars with the other more famous Paul Andrew that\u0026rsquo;s out there.\nKN: But now, you know what you can do, what you can change in terms of your job when you get retirement?\nPA: Yeah, I will design shoes. Just, you know, impersonate him and start doing that job instead.\nKN: OK, cool. Thank you very much. So it was your 100th activity for the community - our podcast yeah?\nPA: Yes, SQL Player 100 community talk.\nKN: Great, great. We will advertise that in that way as well.\nPA: I should have, I should go and celebrate personally, shouldn’t I?\nKN: We can do, haha. Thank you very much, Paul, again for that conversation.\nPA: Thanks Kamil, thanks for having me.\nUseful links Paul\u0026rsquo;s profiles: Twitter | LinkedIn | Blog | GitHub\nRelated events: Data Relay\nPaul\u0026rsquo;s second home: Altius Data\n","date":"2020-05-02T10:27:31Z","image":"/uploads/2020/05/ASF031-SQLDay-Paul-Andrew.jpg","permalink":"/2020/05/asf-031-paul-andrew-interview/","title":"ASF 031: Paul Andrew interview"},{"content":"Out of the box, SSDT covers objects on a database level, like tables, stored procedures, views, functions, etc. That means schema only. In many cases, there is a need to INSERT initial data to a particular table(s). Very often they are called: master data, reference data, lookup data, static data or dictionary. Even though SSDT does not support data - you can take advantage of the post-deployment script and INSERT needed data in there.\nHow to do it many times, how to care of existing data in the target database, when and how to update the data and, finally, how to script those tables from an existing system and easily implement it within SSDT project?\nI will explain in this video.\nLinks Free app: Data Script Writer\n","date":"2020-04-27T22:47:16Z","image":"/uploads/2020/04/ssdt_master-data-video_700.jpg","permalink":"/2020/04/ssdt-scripting-static-data-from-sql-server-video/","title":"SSDT: Scripting static data from SQL Server (video)"},{"content":"In the previous episode, I showed how to deploy Azure Data Factory in a way recommended by Microsoft, which is deployment from adf_publish branch from ARM template.\nHowever, there is another way to build CD process for ADF, directly from JSON files which represent all Data Factory objects.\nIn this video, I will show you the latter method, building the entire process step-by-step, including the fact that we have multiple environments and wherefore isolated services. Therefore, we need to manage all the needed properties, correctly.\n**IMPORTANT NOTICE:**Update @ 02/06/2020:\nSince now, I recommend this extension to Azure DevOps only, if you plan to deploy ADF from code.\nExtension: Deploy Azure Data Factory by SQLPlayer\nIt is free from objects dependencies issue and does all the work in one task. Check it out.\nUpdate @ 02/05/2020:\nAlternatively, you can use Azure PowerShell task in Azure DevOps and use a brand new PowerShell open-source module to publish the whole Azure Data Factory code from your [master] branch or directly from your local machine. The module resolves all pains existed so far in any other solution, including deployment part of objects, deleting objects not existing in the source any longer, stop/start triggers, etc.\nMore details and full documentation with code is in GitHub: https://github.com/SQLPlayer/azure.datafactory.tools\nPeople are still divided on how to deploy ADF:\nLinks The series about ADF on SQLPlayer blog.\nExtension to Azure DevOps I used in this episode.\n","date":"2020-04-01T08:01:13Z","image":"/uploads/2020/04/2020-04-01-video-thumbnail.jpg","permalink":"/2020/04/adf-deployment-from-master-branch-code-json-files/","title":"ADF - Deployment from master branch code (JSON files)"},{"content":" Introduction Benni De Jagere is a Senior Data Insights Consultant with a strong focus on the Microsoft (BI) Stack.\nOn a daily basis, he turns (large amounts) of coffee into insights for customers, and references witty British comedy way too often. Overly enthusiastic about anything data related, he’s trying hard to keep up with all things new and shiny.\nRumour has it that he\u0026rsquo;s also involved with a ragtag band of data enthusiasts, enjoying themselves whilst organising cool community things. They go by the name of \u0026hellip; dataMinds!\nThis talk has taken place during SQL Saturday 2020 in Mechelen, Belgium on 7 March 2020 (Saturday).\nBear in mind that we were in a moment when the corona-virus just started spreading across Europe and 5 days earlier Microsoft made the decision that this year’s MVP Summit will be an online-only / virtual event.\nInterviewer: Kamil Nowinski (T).\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript KN: Hello everyone, today we are sitting in SQL Saturday in Belgium. But the city we’re sitting in is\u0026hellip; Mechelen?\nBJ: Mechelen. The pronunciation is quite OK.\nKN: OK, I’m trying my best. And we are sitting with Benni. Could you introduce yourself?\nBJ: Yes, so I am Benni De Jagere. I am a Belgian Data Platform consultant, doing all sorts of things in the Microsoft Azure Data Platform stack, and on top of that I try to do some things in the community with a user group called dataMinds.\nKN: So I know there is the dataMinds user group and there’s also a conference called dataMinds Connect.\nBJ: Correct. dataMinds does a few things each year but the pinnacle of our events is dataMinds Connect. It’s a two-day conference where we do a pre-con track, which means a day full of sessions by one speaker, going in-depth into a topic ranging from introduction to Databricks and going deeper or Power BI, how to govern it, and all sorts of things. The second day is a conference day where there are six tracks with six sessions each of 60 minutes, so six is a number that comes back a lot. And it is 60-minute sessions by all sorts of different speakers, like we have at SQL Saturdays for instance, but a bit bigger.\nKN: And more commercial?\nBJ: Not more commercial. It is still a user group that drives it, the only difference is that they effectively need to pay to attend, unlike SQL Saturday.\nKN: That’s what I meant, OK, by commercial I mean that it’s not a free event.\nBJ: It’s not a free event, but it is not for profit-driven, meaning that every penny that we keep afterwards we put into different things like today, SQL Saturday.\nKN: So it’s dataMinds Connect. And when does it happen this year?\nBJ: Well, dataMinds Connect will take place on Monday, October 12 and Tuesday, October 13.\nKN: How many people do you expect?\nBJ: Well, last year we had 500 people on the second day, the conference day, and we had 380 people on the pre-con day. And we’re aiming at 50 more each day so 430 on Monday and then 550 on Tuesday.\nKN: That means this is quite a big conference.\nBJ: It’s getting there. The thing is that we don’t want to grow too much at once, because we quite like the venue we’re at right now. It’s a nice building, the insides are good, we love working with those people. And if we grow too fast too soon, the venue cannot handle us anymore and then we have to go venue shopping, which is a hassle.\nKN: OK. What kind of speakers do you have at the conference?\nBJ: It ranges all sorts of things. It goes from people we get from Microsoft, for instance, last year we had Kasper de Jonge from the Power BI CAT Team coming over, Patrick LeBlanc, Adam Saxton, two MVP experts like Ben Weissman, Enrico van de Laar, more experienced speakers as well, and we also try to actively promote getting Belgian speakers, because we are still a user group, so we actively want to get Belgian people there as well. So there is a fair portion of Belgian people, but one of the things that we put an emphasis on, that we try really hard, is the track that we call the newcomer track. And this is a track that we’ve borrowed the idea from DataGrillen, from William Durkin and Ben Weissman. We loved the idea that they did there and it is effectively just people that have never presented at a bigger conference. Inexperienced is a bad term but they are not experienced speakers at all. So we paired them up with a buddy and they help each other, they write up the story, they help them with the slide design, do some dry runs of the talk, and on the day itself they are inside the room as well, making sure that they have a friendly face to look at, that they are not alone in the big room. And we’ve done it for two years past and it is a big success, I’d like to say. Because there are people that have presented in a newcomer track, they’ve effectively felt the rush of presenting in public and they’re doing it still to date.\nKN: Fantastic, a great achievement.\nBJ: It is. We saw the same thing happening with the DataGrillen track and then Data Scotland does it as well and it is fun to see new people coming up, because we don’t want to see the same people every single time again. Some diversity is good and that’s what we’re actively trying to achieve.\nKN: Seeing the same people all the time might be a little bit boring for attendees.\nBJ: It’s fun because you get to know those people more in a deeper way as well, but seeing new people present is also good fun, because they have some different insights and they have a different way of talking. And the main thing is as well is that one of the confusions that we had about the newcomer track is that it’s all going to be basic sessions, it’s all going to be introductory level sessions, so it’s going to be the junior consultants presenting. And we’ve actively negated that confusion over the past years they are all people that are experienced in what they do, they know what you’re talking about, they’re not going level 400 or 500 expert level deep dives\u0026hellip;\nKN But also not 100, yeah?\nBJ: No, They do 300 solid sessions, meaning that they know what they talk about and they do a good job with that.\nKN: They’re very well prepared because they want to do it very well, especially at their first session.\nBJ: They are overly prepared, yeah. But it is also good fun to see the mentors or buddies, or whatever you want to call them, because they get a lot of good fun out of it as well. Because they sacrifice free time for it, they are well aware of it, but they help launch a new speaker as well, which for them is amazing as well.\nKN: You mentioned last year you had Kasper, Adam, Patrick. That’s when the dinner happened, the call happened that I saw yesterday on Twitter?\nBJ: Yeah, the infamous Teams call, yeah.\nKN: Some people maybe, if they use Twitter, they know what we’re talking about.\nBJ: Yeah, but it is good fun. A speaker dinner is intended for all people and all speakers to unwind a bit, to get to know each other and drink if they want to. And it is good fun.\nKN: It’s not only work, it’s also fun.\nOK, tell me exactly what you do for a living.\nBJ: I am a consultant and at this point, the last few years I am a consultant that does a lot of shorter assignments together. Meaning that I don’t have a client where I’m working full time for, I’m doing a few other clients that are shorter assignments or one day a week and then doing those kinds of things.\nKN: Oh, really short assignment.\nBJ: It is running for a longer time but it is one day a week that I spend time for them. And it ranges from doing Power BI development, report developments for a pharmaceutical company, it is helping a public company manage their Power BI premium capacity and helping them tune their Power BI data sets, it is helping another company architect their Azure Data Platform landscape for the coming years. So it’s all sorts of things for each other. It’s fun, it’s challenging but it’s demanding as well because you need to stay on top of your game of all the different things.\nKN: So it’s mainly Power BI but not only?\nBJ: The majority of my time goes to Azure Data Platform and landscaping architecture things. But the thing I like to do a lot at this point right now is Power BI. That’s what I enjoy most at this point.\nKN: So for the last year you’ve been mostly engaged in Power BI?\nBJ: Yeah, all sessions that I’ve presented the last two years of all have been Power BI.\nKN: You know why I was asking about that? You know where I’m going?\nBJ: I don’t know.\nKN: So quite recently, this month, last Sunday, you have been awarded the MVP title. How did you feel?\nBJ: Well, surprised. I can genuinely say that I was surprised because I had reopened my nomination process not that long ago and I hadn’t even gotten a word “we’re working on your application”. So I was genuinely surprised but then the second you read the email, because I was doing things on Sunday in another one of my spare time hobbies, it felt good because it’s a form of recognition for things that you do, and it’s nice to be appreciated.\nKN: So after that Gianluca mentioned on Twitter that he is no longer the tallest MVP among the members, yes?\nBJ: Yes, I’m not sure on how many centimetres I’m taller than him but Gianluca will always be the one with the best taste for coffee. No one will ever take that away from him.\nKN: So tell me how tall you are.\nBJ: Well, in centimetres it’s 2 meters 3 centimetres, and I’m told that in British size it’s 6 foot 7.\nKN: I’m still using the metrics.\nBJ: Metrics is better, it makes more sense. 203 centimetres.\nKN: Next time, when I’m speaking to Gianluca, I will ask him how tall he is. OK, so tell me in terms of MVP: how does this title, this recognition helps you in your work, in your commitment to the community. How does it work for you?\nBJ: Well, the thing is that one of the reasons I gave to Microsoft as well is that as a user group, having someone with an active MVP title helps a lot in making connections with other people and with Microsoft, because it’s easier to make that communication for the first time with them. So that is something that will definitely be helpful for the user group that we’re doing. And I think for me personally well it’s a nice benefit that I get some Azure credits to play around with. That’s always welcome. But I think it’ll allow me to go deeper into some of the topics from Azure Data Platform that I may have not been going into before and then start digging into those as well because those product group interactions that you can do, I think those will be valuable for me as well.\nKN: Are you struggling to read through all those emails that you’re getting? Especially in the Power BI area there’s plenty of them.\nBJ: For now it is the SQL Server that’s been most active for me. But the second I saw it was emails coming in, I made a rule in Outlook that shifts them all to a separate folder that I read when I can. But it is a lot of emails yeah, and I’m not gonna read them all.\nKN: Oh, definitely. That’s almost impossible.\nHow do you prepare for a speech or if you’re preparing your session?\nBJ: I do not like to be over-prepared. I still want some\u0026hellip; improvisation is not the proper word but I still want some fluidity in my sessions. So the main thing that I want to prepare for myself always is making sure that my demos are working, that they do what I want them to do, and the first time I’m doing a session, I make sure that I’m very well prepared for it, but then all the repeats that I do of a session, making sure that I can get the general flow of the session, that I know what the storyline is and that I can build up on that storyline throughout the session. I’m not that kind of person that rehearses every single bullet or every single visual that I have in a PowerPoint, because that limits me in the way that I present. It just doesn’t work for me.\nKN: So do you change your slide or demo at the last moment like last day or something like that?\nBJ: No.\nKN: Never?\nBJ: No.\nKN: So when do you freeze the things?\nBJ: I haven’t done it yet because the sessions that I’ve been presenting have been quite stable as a topic. But yeah, I think it would depend if it’s a breaking change to something, you’d have to change it. But other than that I try to freeze it out two days before, so that my mind has some time to mull over some things, to get some talking points in my head, and then two days before is my set date.\nKN: Yeah, I’m mostly talking about a situation when you rehearse the session and a lot of ideas come to your mind and “maybe, I’ll add this and this, maybe I’ll change the order of the slides or maybe I’ll add something to the demo”.\nBJ: The thing is that I do a session about Power BI dataflows as well and I’ve had it a few times already that they’ve done some changes that break some things that I noticed an hour before the session, that I’m doing a final check of a demo and “ahh, they changed this”. And then I’m glad that I’m doing it an hour beforehand and not noticing it during the session. Because I’ve had that experience as well, not in community presentations but at customer presentations. It’s like “the buttons should be there in the corner but it’s gone so they moved it somewhere – please allow me to find it”.\nKN: So, about this newcomer track: I know, we all know that is quite a challenge to encourage new people to start speaking. So how can you encourage people to start doing that?\nBJ: Well, I’ve found that the way that you can get people to convince them is by face-to-face interaction, just explaining to them one on one what it is, what they can do, what they need to prepare for it, and what it’s going to be like. Like the general approach and sending an email to 20 people, like: “hey, do you want to do a newcomer track?”, that typically does not get that much response. And it is engaging with those people one-on-one that helps the most, I think. And we’ve tried it over the past years and it’s worked out quite well.\nKN: Because there’s live interaction, you can answer for the new questions or explain how it works.\nBJ: For instance we had someone present a user group session for us in February, and during the introduction to that talk I explained dataMinds Connect what we were going to do and the newcomer track as well. And then afterwards he said like “yeah, I actually like to do that, the newcomer track. Can I still do that because I’ve done this user group session?”. And yes, you can still do that. The only limitation we put on is the fact that you cannot have presented at another technical conference or another bigger event. So anything like customer presentations or user group sessions – they don’t matter.\nKN: And we know that the IT market specifically needs a lot of people and a lot of specialists. So what kind of hints would you like to give young people to start working on the IT market? How to start, basically.\nBJ: I’m a firm believer in functional, non-technical soft skills. I think that those are the things that help people advance more firmly in an IT career. But it is quite clear that data and data-related jobs are going to be key in the future. It is quite clear. And something that I still regret of not having done more in the past is having a more statistical, mathematical base for allowing me to do those more data science kind of things. Analytics, like the things we do in business intelligence, those I can do perfectly but I’m not the kind of person that’s going to draw out neural networks or those kinds of things. So if you were to talk about data engineer and data scientist, I’m clearly on the data engineer side. And then, I would have loved to be doing more of a data science side as well but you have to make decisions sometimes and you have to choose what you’re going to do. And for me right now I’ve said: I’m going to go deeper into data engineering and then I’ll leave data science up to other people, because data scientists need me as well because they need their data to be prepared properly.\nKN: Yeah, absolutely. There’s still a lot of cooperation.\nBJ: The thing is that, focus on those mathematical skills, the statistical base, that’s going to be good for anything you do with data. And then, work on data manipulation, is it SQL, is it Python, whatever, but work on those kinds of skills. And I think that they will get you very far in an IT data-related career.\nKN: Yeah, also transformation data. Now you can do it in Power BI as well with dataflows. How do you think it works? I mean, you are much closer to Power BI and dataflows.\nBJ: I hate it and I love it. The concept of dataflows is excellent. I really love the concept. The thing is that for it to properly work in a way that I myself as a data engineer, an ETL guy, for that to work you’re pretty much stuck to premium. So dataflows is not a premium product. If you talk to any of the Power BI CAT Team members they’ll hammer that into you: Data Flow is not premium only. But you need some of the key features, like in Lake computing and then linked entities, and they require premium capacity. And they allow you to make a better daily lineage across your different entities and for it to work the way it should be. But dataflows in general, I’m positioning it these days with customers like, for instance if they have some master data that needs to be managed or we have an Excel file with a mapping or something. That’s where I’m positioning dataflows. I’m like: OK, use dataflows, land it somewhere in Data Lake Storage and then we’ll pick it up with Data Factory or Databricks to use that mapping file in our own thing. So you’re effectively just making them the responsible ones for their own master data or mapping files.\nKN: They know the data. They know the quality of the data.\nBJ: Yeah, we just tell them: “have it there by that hour of night and then we’ll pick it up and do the thing that we need to do with it”. And that works quite well. And then, the other way I’ve positioned it is allowing certain key figures in an organization that have some technical skills, I’m not saying IT users but people that have been working with Power BI, people that know how to work those things, have them build small prototyping entities.\nKN: Yeah, I wanted to ask about POC [Proof of Concept].\nBJ: Not that much POC but it can be used for that as well, but also for, let’s say that you have a Data Warehouse or Data Lake and there is something that’s missing in there, according to business. And your data engineers don’t have capacity for it, they cannot build it. I have used it in the past to have those people build out a small thing for themselves, have them build it the way it should be, the way it should be working and then, after they’ve had some collaboration with IT, the way it should be and how it effectively should be working. Just transition that and do change of ownership to IT and then they incorporate that into the entire Data Lake flow. So that’s how I’ve positioned it in the past as well. And that worked pretty well, but those key users had decent knowledge of what they were doing, and I do think that that is key for a setup like that to work. But it is an exciting tool, it is showing potential, but the pro version has limits if you want to build an entire ETL flow for that. But on the other hand, you shouldn’t be building an entire ETL flow in dataflows. For many reasons. We’re all data professionals, we’re all BI engineers and we know that some things need to be done properly instead of in a self service tool. But there are options to do it.\nKN: Performance-wise as well?\nBJ: Performance as well. For now incremental refresh only works in premium in dataflows. So you always have to load the entire data set, which doesn’t always work that well. And some sources like CSVs have not that good of a performance if you load them in with dataflows. So there’s clear differences, but yeah, like I mentioned, it is a good tool for business people to build prototypes in or build gaps into some things, but you’re not building enterprise data lakes with dataflows. That’s not the way you should position it.\nKN: Yeah, that’s what I thought. It’s perfect to build something at the beginning, it’s perfect to prove that it might work with that idea, with that set of transformations, but for the bigger solution, you need to just transform the solution basically and use different tools. What about the Common Data Model in Power BI? That’s a cool feature, yeah?\nBJ: It is cool. I want to do more with it, though. For now it’s just landing things in there, making sure that it’s just properly formed, that it’s been usable, but the main thing in there as well is that the data is stored in CSV.\nKN: Do you know why?\nBJ: I don’t know why. That’s one of the product group interactions that I want to do in the future. But it is stored in CSV files, which is OK, but that just underlines the previous statement that I made that you’re not going to build enterprise-like stores and their data lakes in there. But it is a cool feature, especially one of the things that I’m going to be working on in the next few months with a few of my Dynamics 365 colleagues is using data that we land from dataflows into Common Data Model to be picked up by Common Data Service by Dynamics 365 and the other way around, so that we can tie those two together. That’s one of the things that I’ll be doing in the next months.\nKN: Oh, pretty interesting.\nBJ: Yeah, I want to try it.\nKN: But still, under the hood, there’s Azure Data Lake Storage, yeah? So it’s very effective, very performant in terms of read and write, so even CSV shouldn’t be so bad. Obviously, there’s some other different formats\u0026hellip;\nBJ: If I can get Parquet files as a Common Data Model storage, I’d take it with both hands.\nKN: Maybe in the next version we will have an option.\nBJ: We can only hope.\nKN: What about your work-life balance. How can you cope with that?\nBJ: I don’t have a balance I think. The last few months have been wild for me. But I’m a consultant and in general the agreements that I have with customers and my employer is get it done, and if you can do it in 7 hours it’s okay, if you can do it in 9 hours, well\u0026hellip; So I can do some balancing and some flexible things there, for instance when I need to leave at 4:30 to make it to a venue for a user group meeting in time. They’re not that fussy about it. I just need to make sure that the job is done. But other than that - we do quite a lot of user group meetings throughout the year as well. We do SQL Saturday, we do Global AI Bootcamps and AI Nights, we do a two-day conference, so there’s a lot of time that gets into that as well. But it is fun to do because otherwise I would not be doing it. But other than that, I do try at least two or three times a week to go out to karate, try to practice.\nKN: Yeah, exactly, that’s my next question. I’ve seen on your Twitter profile\u0026hellip;\nBJ: Full-contact karate, yeah. It is an entirely different thing. But also it’s a sport that requires a lot of respect for each other, and I really appreciate that. For instance back in the day, when I still used to compete, I don’t do it anymore these days, you effectively get on the mat, you give each other a very hard time. You beat each other, you kick each other. But the second it’s over, you get off the mat and you hug the other guy and you start cracking jokes. There’s such a deep form of respect for each other that it is genuinely a good thing to do. And yes, you’ll get bruises or yes you’ll have some pains afterwards but yeah it is still a fun sport to do.\nKN: I’m just wondering, you know, who might be that brave to stand in front of you and fight with you.\nBJ: Well, a lot of people. They think it’s a challenge. And the general thing is that I’m quite tall and I’m not the lightest one as well, so they can work on speed to get past me. I’m not that fast, so if they’re fast and they can do Muhammad Ali like “float like a butterfly sting like a bee”, that works for them as well. But yeah, I land fewer kicks and I land fewer punches but the ones that do land, they come across well. But like I said, it is all good fun. We have these things called 50 people kumite, which means that we do 50 rounds of sparring against all people that are there. A hundred people get together, we do 50 rounds of sparring together. And it’s 50 times 2 minutes after each other that you do some sparring matches. And there’s two breaks in between. So you die 27 times at least, but it’s a way of the sport, I think. If you don’t do that kind of sport, it is difficult to understand, but for us it’s a lot of fun. But it is also a matter of respect because sparing is not beating the pulp out of each other. It is making sure that you try out something new and yeah, you hit each other but you don’t hit each other to effectively break your ribs or something.\nKN: That’s interesting. So how often do you attend?\nBJ: I try to get to practice two to three times a week. Sometimes it works, sometimes it won’t. If I’m travelling abroad it’s not but in general it works out quite well. But for me it’s a release mechanism to just relax, to beat a punching bag or to just let everything go.\nKN: Do something physical, not just mental.\nBJ: Yeah, I need it, I really do. I’ve had some periods of injury in the past, I couldn’t go train for two months and those were two months that I was really antsy and yeah.\nKN: And you also mentioned that you are a coffee addict. Are you? How many coffees do you drink a day?\nBJ: Yes, it depends really. I’m trying to cut back. The thing is that I drink a lot of coffee but I drink it on a low strength and I take half cups, meaning that I still drink a lot of coffee, because it forces me to get up to walk to the coffee machine, to take a lap around the office and not sit behind a desk all day. But six cups a day is not an oddity. And if I’m working towards deadlines or if it’s days like today where I haven’t slept that much, it’s more than six, definitely.\nKN: It was because of yesterday’s dinner.\nBJ: Well, yeah. I got back to the hotel at around midnight and I was here at 7:00 so not that much sleep.\nKN: So your hobby is karate\u0026hellip; How do you say it?\nBJ: Full contact karate. Shinkyokushin.\nKN: It’s a completely new area for me.\nBJ: Yeah, in Europe it’s quite big. The countries like Poland and Lithuania and Bulgaria and Hungary, they’re really big in that sport. Here in Belgium it’s a hobby. The countries that I’ve mentioned before, they’ve got professional athletes that get paid to train, paid to compete. But out here in Belgium and the Netherlands and France as well it’s something we do after work. So we get home from work, we go train, so as for level, we’re not that high level as the countries like Poland in Lithuania. They are the absolute top in European rankings, but it is still Japan that is the absolute Lord and Master of the entire thing. They start training younglings from 3 to 4 years old and they train really hard a lot. And they also have professional athletes. They start training them in the techniques etc. and not starting to train them to be killing machines from age 4. But Japan is also really protective of their competitors, underage competitors. They are very heavily protected up until 18, and then from the second they get 18, it is full contact as we all do it, meaning no protection. Just groin protection.\nKN: So how much do you travel across Europe or across the world?\nBJ: The last months have been a bit more, because I’ve been speaking more. But I try to limit it to one European trip a month. That’s what I try to limit myself to.\nKN: I also try to limit you know like up to one even per month averagely but it doesn’t always work.\nBJ: Yeah, February was\u0026hellip; I’ve said yes to a few things in Belgium that were happening, organised a few things myself, and then some other international things came along as well and I said: “yeah I do want to do that” because I got a selection to SQLBits last Sunday as well. They emailed me and said that “Do you want to come present?”, and I’m like “Yeah, well, I do”. I’m travelling already in March in April but it’s SQLBits. You’re not gonna say no to SQLBits. So I try to limit myself but it does not always work.\nKN: So obviously you’re going to SQLBits?\nBJ: Yeah.\nKN: SQLDay this year as well.\nBJ: I am going to SQLDay as well, in Poland, yeah. It will be my first time in Poland.\nKN: Fantastic, great to have you there. So what do you think about what’s happening due to this corona-virus and MVP Summit in the next week?\nBJ: Well, I get why they do it. Especially because the state of Washington has been hit hard in America, that’s why I really understand why they’re doing it. For me this year is not that big of a deal because I wouldn’t be able to attend anyway. They’ve notified me on March 1st and then it was a bit too short notice to get a trip to Seattle and hotels and all those things to get it sorted out, and I already have customer visits planned that week.\nKN: Yeah, obviously, you have just been awarded, so it’s too late. I’m sorry, I mentioned that to you before that in my mind I thought that you had been already an MVP for years.\nBJ: No, but we’ve got the dates for 2021 already and I’ve blocked a mini schedule and I want to travel there if I can, so yeah. But I definitely don’t get to go this year. For me personally it’s not that big of a deal that it’s been cancelled but I understand why they do it. Same reason why they’ve locked down campus, that they’re asking everyone to work from home. I really get it.\nKN: Yeah, absolutely, it makes sense. And to be more precise, Microsoft hasn’t cancelled that event. It’s just been moved online.\nBJ: I’ve yet to see how it effectively will work out but the thing is as well if I have to be at a client during the day, and if there’s a session at 3:00 a.m., I have to be reasonable and make sure that I can do my client work as well because they expect me to do some work. So I’ve yet to see what it’s going to be, what the schedule is going to be like, if there are going to be recordings or video on demand or something but I’ll see when it gets there.\nKN: Recordings are quite good but it doesn’t always work after the event.\nBJ: It’s not the same.\nKN: It’s not easy to find the time to watch the video again.\nBJ: But the thing is that the MVP Summit, the content alone is great, but the main reason you go there is the interaction with other people. It is like this very big reunion of all people coming together and that’s the main reason why I would want to attend there as well.\nKN: So your session today was about Power BI troubleshooting your Power BI report performance. Could you tell us a few things about what you presented?\nBJ: Well, I’ll start off with a sentence with what I didn’t present. It was not a DAX deep dive performance session. I leave that up to Marco Russo, Alberto Ferrari, Kasper, those kinds of people that are DAX experts. I go to great lengths to avoid having to write complex DAX, which means that I model things. I star schema all the things, which is the baseline of the entire session. It is fix the things in the model to avoid having to do complex things afterwards and that is the main reason how you can get your performance to be better. Because the things I run into at clients is that they dump Excel files in there and there we’re starting to do BI-directional filtering all over the place and they’re starting to write complex DAX with things that could be done probably more efficiently. So a star schema is the base of everything in Power BI. That’s my opinion on it and I’ve found many people to share that opinion. I’m aware that star schema is a difficult thing if you’ve not learned it before from a BI perspective or something, data warehousing, but I do feel that it’s a key skill to learn if you’re serious about Power BI. But other than that it is getting started with the tools that Power BI has itself, to performance analyzer, how you can see what’s effectively running slow, what can you do about it, and then using a few community tools like DAX Studio, VertiPaq Analyzer, Tabular Editor, Power BI Helper, those kind of tools to start pinpointing your way through what be improved upon. So typically what I do is when someone comes to me and says: “my report is slow, fix it, please”, then what I typically do is I try to isolate the query that runs slow, or the queries that run slow, then I open up VertiPaq Analyzer, well DAX Studio to see what’s effectively in there and how its distributed between the different engines in Power BI, open up VertiPaq Analyzer to see what the model internals are looking like, and then go from there. And that’s usually how I get started on working on those things. And if it’s effectively well-modeled and the DAX has to be fixed, then I’ll do that, but usually I go for the model first.\nKN: So basically the rule is still valid, the rule that says that 80% of the problems comes from the model, the design.\nBJ; Yes, the design, the data quality, the data cleansing.\nKN: It’s still true even for Power BI, where the whole operation is in the memory.\nBJ: The thing is that I also see things in reports that people have built that they’re fixing things in calculated columns, like for instance I want to have an age bucket. Is it age 30 to 50, is it age 50 to 70, and they start doing that in calculated columns in Power BI, which is\u0026hellip; Yes, OK, you can do it but it would be way more efficient to have it in Power Query, where you can fix it and it’s been optimized in the model, or even if you can, put it in the Data Warehouse, just to do it as soon as possible in your process, so it can be reused throughout all the reports that you’re doing. Because if you’re doing it in Power Query alone and you have to use that same age bucket somewhere else, you have to remake it. So putting it as soon as possible in the process works. And that’s what data warehousing is about. That’s what we’ve been telling for a few years.\nKN: That’s cool. And also that’s good to know that some tools exist to do that diagnosis.\nBJ: There are many community tools out there, and they are of great help, yes.\nKN: OK, Benni, thank you very much for this conversation. And at the end, could you tell us where we can find you, where people can find you, follow you?\nBJ: Well, they can find me on Twitter and LinkedIn. It’s mainly just my first name and my last name – Benni De Jagere. But good luck on trying to spell that if you’re not Flemish.\nKN: Definitely! I will put the links under the transcription. OK, so LinkedIn and Twitter. Do you still have your blog?\nBJ: I still do but it’s not been updated for 5 years. That’s one of the things that I’ve been telling myself for the past year to start doing again, but work-life balance is getting in the way.\nKN: I definitely understand it, you are already very, very busy doing all this stuff. So yeah, I appreciate that, OK.\nBJ: It’s fun, so that’s the reason why I do it.\nKN: Yeah, exactly. Thank you very much again!\nBJ: Thank you!\nUseful links Benni\u0026rsquo;s profiles: Twitter | LinkedIn\nRelated events: dataMinds Connect | DataGrillen\n","date":"2020-03-27T17:25:35Z","image":"/uploads/2020/03/ASF030-SQLSaturday-with-Benni-de-Jagere-1200.jpg","permalink":"/2020/03/asf-030-benni-de-jagere-interview/","title":"ASF 030: Benni De Jagere interview"},{"content":"Building CI/CD process for Azure Data Factory is not quite straightforward. Furthermore, there are a few different methods of doing that. Before we begin doing that, we must set up git repository associated with ADF, understand what\u0026rsquo;s the differences between save and publish, and why new branch appears.\nThe basic knowledge of ARM templates also would be beneficial.\nDuring this video, I will explain one of the methods recommended by Microsoft.\nhttps://youtu.be/F8kF9h86_XE?list=PLcW0jJygu-2dgBMbtYXCOC2Dx-1KLINEe\n","date":"2020-03-14T07:10:02Z","image":"/uploads/2020/03/2020-03-13-Videos-Thumbnail.jpg","permalink":"/2020/03/adf-continuous-integration-deployment-with-azure-devops/","title":"ADF - Continuous Integration \u0026 Deployment with Azure DevOps"},{"content":" Introduction Tomaž Kastrun is BI \u0026amp; DEV developer, data analyst \u0026amp; data science.\nWith more than 15 years of experiences in the field of databases, business warehouses and development, with a focus on T-SQL programming and query optimization. His focus is also data mining, statistics and research. He has been working with Microsoft SQL Server since version 2000.\nHe is Microsoft MVP, Microsoft Certified Professional and Microsoft Trainer.\nTomaž is a blogger, author of many articles, speaker at various community and Microsoft events and an avid coffee drinker.\nThis talk has taken place after SQL Saturday 2019 in Ljubljana, Slovenia on 14 December 2019 (Saturday).\nInterviewer: Kamil Nowinski (T).\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript KN: So today we are sitting at the venue during the SQL Saturday conference in Slovenia with…\nTK: Tomaž Kaštrun.\nKN: It’s very hard to pronounce your name.\nTK: Yeah, Slovenian names are hard to pronounce but your pronunciation is correct.\nKN: Thanks! So where do you live?\nTK: I live here in Ljubljana, where the venue for the SQL Saturday Slovenia is happening. So this is my hometown and welcome to my hometown!\nKN: Thank you very much! I know that you are one of the organizers of this conference. Could you tell us a little bit more about the conference itself and what is your role as the organizer?\nTK: So SQL Saturday Slovenia this year we have the 7th iteration, so we started in 2013, I joined in 2015 I think, so literally we are full and I was the last one to join, and we usually just have each their own task that we need to cover and that’s it. At the event, yeah, all the usual stuff, so accepting the people, accepting the attendees, taking care of the speakers, we give great attention to every detail, to all of the speakers because they come from abroad and they’re tired, stuff like that, so we order this, we do this, coming a step forward closer to them, and this really helps them a lot as well.\nKN: Yeah, it’s quite a local conference but it’s quite huge and very popular. One of the most important during the year I think. You have speakers from around the world.\nTK: We are definitely very proud of that. We are taking quite huge pride in that. And this is something, you know, if you go abroad and if you visit all the people, get in touch with them, all the speakers, they will definitely come and sort of return the favour, come to your place. So it became a tradition. Since I’m the last one to join, I think Mladen and Dejan, and Matija, they were the first to start this attention to details and bringing a lot of people from abroad, and now this year is just amazing. And so it was last year. So everything is really amazing and we try to improve to get better. Not bigger but better in terms of bringing different audience, bringing different speakers and choosing the right\u0026hellip;\nKN: Keep the quality of sessions.\nTK: Yes, the variety of sessions.\nKN: The size of the conference is not as important as the quality of the sessions.\nTK: So as long as we have speakers happy and attendees happy, I think this is the best recipe.\nKN: Great, and I think you are doing a great job here, hospitality is fantastic.\nTK: This is what we really try to keep the level up.\nKN: So what do you do for a living?\nTK: So I’m a BI developer/data scientist and I basically do a lot of stuff with machine learning, data mining, statistics, statistical analysis, and a lot of stuff with SQL, with cloud computing, Azure, stuff like that.\nKN: A great mix.\nTK: Yeah, it’s a hybrid.\nKN: So BI but probably these days with more modern BI? In the cloud and with all the new services and also I’ve seen you speaking on the topic like Machine Learning for example, R language, that kind of things. So, about R language, could you tell our audience how to start or how difficult the R language is to learn and when you can use it, in which cases?\nTK: There is the competition called Advent of Code going on right now, so if you Google adventofcode.com, every day you get sort of like a task you need to solve in any kind of programming language. So this year I decided to do it in R, which turned out sometimes a little bit problematic, but it goes so, which means that R is a statistical programming language with more stress on statistics than in programming but yeah, I’ve been using R for quite a long time, as well as Python. Once you start using it, you get fluent in it, for those who want to start and learn Python or R, or both, I would definitely suggest using it and learning it by cases, examples and stuff like this.\nKN: Both of them?\nTK: Well, one of them, both of them, many of them, it doesn’t really matter. The best way to practice is to have some examples, samples and\u0026hellip;\nKN: Because R language is like you said, more for statistical point of view, so this way it’s more often used by mathematicians.\nTK: Yes, statisticians, mathematicians. But you can also do programming. It’s not the size of C++, C# or Python, but also you can do some programming.\nKN: OK, and if someone has not started using either R or Python, which language would be easier to learn or use?\nTK: Really depends on the background so if you have more programming languages, you would definitely prefer Python over R. If you don’t have that much of programming background, you would prefer R over Python. But in reality, I think the modern data scientists are people dealing with statistics, dealing with machine learning, data mining, they ought or they should or they must use different and more languages. So R, Python, SAS, I don’t know, you name it. Julia, Knime, they all use several different languages. Because one language might give you the benefit for this algorithm, the other one for that algorithm, so it really just depends.\nKN: It depends on your scenario or case. OK, let’s talk a little bit about machine learning. Could you explain a little bit what machine learning is?\nTK: As the term suggests, you let the machine learn on something and then execute something. So to sort of give it a different name, we used to call it statistics, then we used to call it data mining, and now all of a sudden it’s called machine learning, which essentially is, you know, it’s not the same but there are huge similarities in terms of the same algorithms, so what it does it just takes, for instances you take some data and based on this data, which we call training data, you train a model. A model is just sort of a presentation of a function and this function basically gives you the next best value. We call it, let’s say, prediction. So this is essentially what it does. You leave some data set, let machine learn itself based on this, and then it produces some sort of result, which is a function, which is a model. And then, based on new data that basically comes in, you try to predict what the model that you’ve trained upon tries to tell you what the next value will be. Of course, there are even other processes behind. You need to test, you need to validate, stuff like that. But essentially this is what it is.\nKN: It already sounds a bit complex. For people who don’t know this.\nTK: Yeah, but otherwise I’d say if you love it, you don’t find it that difficult. Some things are difficult of course. It’s hard to explain an unexplainable. The thing is you just need to sit down and just investigate and learn. You just need to have this continuum. Just don’t give up after five minutes if you don’t get the right result. And this is also something modern data scientists usually forget about. After five minutes: “ahh, the model doesn’t program, that’s crap”. You need tweaking, attention to details, stuff like that. This is essentially what machine learning is. And there is another perspective of that a lot of data scientists think “yeah, this is the sexiest job, yeah, I’ll do a lot of programming, a lot of testing”, where in reality it’s like 80% is just collecting the data\u0026hellip;\nKN: Exploring the data and then cleaning the data.\nTK: Exactly, so this is the problem with it. The skewed perspective of this machine learning world.\nKN: The data science role or position is quite popular right now. What do you think, is this the right direction, calling many of the roles “data science” or this should be called differently?\nTK: I think the data engineer\u0026hellip; So the roles have changed over the years and you would have like DBAs, developers, stuff like that, so nowadays you have all this data science, machine learning has been injected in all these aspects, which means that the modern DBA should also know at least some Python, at least some R, should know how to explore the data setting using Power BI, should know how to use Azure functions\u0026hellip; All this mesh of different technologies, different languages, this is something that the technology is basically pushing us toward, that we need to explore.\nKN: Yeah, exactly, it’s quite hard these days to understand. I mean not to understand, but to stay up to date with all this technology. Basically you must know maybe not all of them but much more than in the previous years.\nTK: In terms of different technologies, sometimes you just need to say “OK, that’s it, I’m not going further because I want to investigate what I’ve learned and try to deepen the knowledge”, because if you’re just trying to catch all these new technologies, basically you run out of energy, you run out of talent and essentially at the end you just have some idea how the technology works but you don’t have any in-depth knowledge of what a particular algorithm does, what a particular language does.\nKN: And for people who are trying to understand or learning about machine learning - how to understand the three types of machine learning algorithms like supervised, unsupervised, reinforcement?\nTK: I always give some similarities to supervised/unsupervised, so imagine, since you have a daughter, imagine telling your daughter that “you should go clean your room” and you give her two buckets and there are dolls and cars on the floor, and you just tell her “clean the floor and clean your room”. And your daughter would be exploring those different toys and based on the similarities of the attributes of the different toys. So let’s say dolls have hair, legs and stuff like that whereas the cars have wheels and stuff like that. She would be exploring herself which would be rather unsupervised. In terms of the supervised learning, you would tell your daughter “put the dolls in bucket #1 and all the cars in bucket #2”, so this is the difference between supervised and unsupervised learning. We also call it directed or undirected learning because with unsupervised or undirected you are not giving any directions where the algorithm needs to go or the exploration of the data or the discovery of the data should go. We don’t know, at this moment we don’t know, let’s explore it. Whereas in supervised, directed you usually know where you want to go.\nKN: That’s a really nice explanation. For sure I will remember that now.\nTK: And with the reinforcement learning I think it’s relatively old but people have kind of been scared of it, not sure, but if you go to ODSC conferences, they will tell you they’re using it for auto-piloting cars and stuff like that. So essentially the idea is, once you go in the right direction, you get awarded, vs if you go in the wrong direction, you get penalized. So this is the essence of the algorithms.\nKN: How to start with machine learning if someone is interested to get that data science role in the future?\nTK: This question is kind of hard to answer because I get a lot of these questions and I have an easy answer because I studied it at the faculty but the problem is if somebody new wants to start, I always say: either explore some problem, go to Kaggle, download the dataset, go find a UCI where they store all the open data sets, and just start exploring, and by learning and doing. Of course, if you can take some classes at the university, that’s even better. There are also some webinars, online classes, books available. So yeah, these are great sources, but in reality, I would say that the best thing is always when you’re in an interaction with a person, with a professor, with somebody who has you know the knowledge, it’s much easier to get on board. So definitely, if you are learning it in a supervised environment, let’s put it in this perspective, then your chances for success should be higher than if you’re exploring it by yourself. I mean, you would succeed fast enough.\nKN: And from a technology perspective, what kind of tools, what can help you? Machine Learning Studio, something else?\nTK: So there is Azure Machine Learning Studio, since you mentioned it. You can use that, which is quite easy. It’s based on Python SDK. You basically just throw in data and there is a set of algorithms that just basically does all the different iterations, goes through different algorithms and basically outputs you the result. There are also some other tools like R Studio, Python IDE. We can just explore that. If you’re in a corporate environment, there are also tools available for that as well, so IBM for instance, SAS which is famous, SAS Institute from the United States. There is Knime, which is a Swiss or German-based, not really sure, company. You get a lot of that. And now with the emergence of different technologies and different companies, we have even more platforms to do machine learning.\nKN: And I think I should provide some links at the end of this conversation in the blog post.\nKN: OK, so what do you think about the MVP these days?\nTK: I’ll go on the other side and answer that the thing from my perspective is that people don’t respect the community work nowadays as they did twenty years ago.\nKN: You mean people like the audience or people who have the MVP?\nTK: In general, so let’s say IT people. Since they know that you can go to Stack Overflow, that you go to YouTube and stuff like that, but in reality, they don’t understand that in the background there is a community. Stack Overflow is a community, YouTube is a community, events like today SQL Saturday Slovenia, that’s a community event. And people forget about that. So they don’t respect that. And there’s the problem that a lot of developers, they just say “yeah, I’m just going to learn it”. “Yeah, who cares about this meetup, who cares about stuff like that”. No, no, no! That’s the wrong perspective. You should commit to that, try to learn something, try to share something, this is what basically builds the community. Since we have this plurality of books and audios and videos and stuff like that nobody really cares, which is wrong because people in behind who create, who are creators of all this material. They should basically be awarded for that, because a lot of people do this for free. And from time to time it’s really hard. I do respect all the MVPs and all their work that they do, it’s really amazing, but on the other hand I kind of, you know, I would really hope and wish that people in the tech world really respected the community work. It’s basically our job to push this, to be sort of like evangelists, leaders.\nKN: Show people that it’s really worth, it might basically change your life, professional life.\nTK: Yeah, so coming back to the MVP award, I think this is a great, great recognition and I’m proud of it. To be honest, I’m proud of it to be recognized as an MVP. And I know it’s hard work.\nKN: Exactly. My next question is going to be “how often do you travel?”, across Europe or worldwide.\nTK: I had a little bit of a pause and break in the last half a year or so, but usually I do 15–20 travels. So SQL Saturdays, different events, different conferences. I try also to introduce Microsoft technology to totally different communities that are related to IT or different conferences that are not necessarily Microsoft-oriented, so yeah, this is usually what I do. I present Microsoft with their technologies, solutions and stuff like that. So yeah, I’d say 15–20 plus blog posts, all the writings\u0026hellip;\nKN: Which also consumes a lot of time.\nTK: Yes. And the tech world also forgets that a lot of community leaders, they do this out of their free time.\nKN: And for free. Like SQL Saturday. All the speakers are doing it for free. Out of curiosity, how long it does it take you to write one blog post?\nTK: That really depends because sometimes you just get an idea and you want to put it out of your head, so it’s just like, I don’t know, 3 hours and sometimes it’s just trouble with it, like two weeks, three weeks and then you decide “OK, maybe I just should not get married to that idea”, so it really depends. But yeah, at least 3 hours. The only rule I really stick to is you know, if you feel doing it, do it. It should come from the heart.\nKN: How do you prepare yourself for a speech/session?\nTK: That’s a funny thing. Usually, I walk in my apartment, talking to myself. Then I envision all the material that I prepare and then I sort of do a dry test with the code and again talk to myself, describing the code and blah-blah-blah. So basically, I’m talking to myself, that’s how I prepare.\nKN: OK, so what is first, you just look for a topic and the content, talk to yourself and prepare some slides and demos?\nTK: I usually have too many ideas and too little time, so this is the problem. Once I decide on one, then I just start collecting the ideas, the material, testing, so at the end when I’m 70% finished or satisfied, I’ll just start to do little updates, incremental updates. And essentially, I’ve never done one session twice, to be honest. So every time I try to add a change or correct the code, change the slide, add this, remove the parts that don’t work, just stuff like that, so essentially, even if it’s the same topic or literally the same presentation, it’s not the same. At least 10–15% is changed.\nKN: Ah, I thought that you create a completely new session. Because obviously you can change slightly because you have experience after your first presentation in front of an audience, but then there’s a lot of people if you travel across Europe, there’s a lot of people who haven’t heard your session.\nTK: Yeah, to be honest, I don’t really\u0026hellip; From that perspective OK, but I don’t really consider that a problem, so I just go through sitting at home and “OK, I’ll remove this, I’ll add this, I’ll remove that, I’ll skip this code, I’ll add this one”.\nKN: So every time you tweak your session.\nTK: Exactly. So it’s like it’s unfinished work to be honest.\nKN: So if you do that kind of things, do you change your slides at the last minute?\nTK: No, this is something I’ve learned the hard way so I don’t do that. Again, ever.\nKN: What kind of conference would you recommend to people?\nTK: Depends on the topics that they are interested in, but in general I would say the conferences where you could speak with the presenters or with the companies are the best ones, because you basically get the knowledge from the field, the insights from the people, and in my personal opinion this is probably the best way of networking, getting to know people and stuff like that.\nKN: Not only knowledge, but you can extend your network.\nTK: Yes, and also again, speaking with a professional who has invested a lot of time preparing a particular slide, a particular presentation should also be considered.\nKN: So what kind of hints would you give to young people if they want to start working in the IT market if they haven’t started working there yet?\nTK: First of all you need to listen to the heart and if it doesn’t sound right, then just don’t do it. If it sounds right, then yeah, you are basically on the right path. And this is the only real advice that I usually give, also for instance to my daughter. Listen to your heart. If you’re doing it out of passion, then you’re doing it right. Because Steve Jobs once said: “once you’re at the end of the road or at the solution, then you realize how to connect the dots back to the start”. Once you are on this path, you don’t really know. You feel if you’re going in the right direction, you get hints, you get some semi results but only when you arrive at the end, you can connect the dots. So basically listen to your heart.\nKN: You need to feel it, you need to love it, and that helps a lot. How did you start your journey?\nTK: Oh, I don’t know. I remember my mum when I was probably like 6–7 years old. She introduced me to a course on programming with I think it was like Spectrum ZX and we were\u0026hellip; Honestly, I don’t remember but I was like 7 years old, so this is how I got introduced into this big world of passion.\nKN: So that was your first contact with a computer. And what was the first connect or what was the first job when you started?\nTK: I think all the next jobs were it was permanent or it was just you know some summer job or whatever had to do something with computers. Honestly, I don’t remember when was the first one, but I can say it all had some relations to computers.\nKN: They all have something in common, computers and IT. OK and the last question: what about your work/life balance? You do all these things for the community\u0026hellip;\nTK: Don’t go there, come on\u0026hellip;\nKN: We all have the same, I would say, maybe not a problem but “challenge”.\nTK: Yeah, that’s a nice perspective. It is a challenge balancing all the community work, all the things that basically drive you, and all of the work that you need to do plus personal, private life. So it is a balance. And if you are mastering it, then it’s easier. If you have problems, then it’s not that easy. So basically yeah. But I think once you realize it, it’s much easier that “OK, I have a passion for that”, that instead of sitting in front of the television, you code something or play with technology. But you know, I know you’ve been there, I’ve been there, we are all basically in the same situation, so it’s hard to balance both, yeah. And there is no formula to that.\nKN: No formula, no recipe. OK, thank you very much Tomaž for this conversation.\nTK: Thank you for having me and I wish you a pleasant stay in Slovenia and at the SQL Saturday Slovenia.\nKN: Thank you very much.\nTK: And thanks for being a speaker and coming here. It was really nice having you.\nKN: Thank you for having me and for this great and fantastic hospitality here. I really love coming back here over and over again. And this year being with my daughter.\nTK: Yeah, she’s beautiful. Thank you!\nKN: Thanks!\nSylvia: Tomaz, thank you very much for your kind words!\nUseful links Tomaz\u0026rsquo;s profiles: Twitter | LinkedIn\nTomaz\u0026rsquo;s blog: Blog\nTomaz\u0026rsquo;s book: SQL Server 2017 Machine Learning Services with R: Data exploration, modelling, and advanced analytics\nRelated events: SQL Saturday Ljubljana\n","date":"2020-02-28T11:07:56Z","image":"/uploads/2020/02/ASF029-SQLSaturday-with-Tomaz-Kastrun-1200.jpg","permalink":"/2020/02/asf-029-tomaz-kastrun-interview/","title":"ASF 029: Tomaz Kastrun interview"},{"content":"Have you worked with ADF yet? Did you configure the GIT code repository to automatically upload all changes to having your own isolated branch during development? If not yet, in this video I will show you how to do it. Finally, we will build a simple pipeline that copies files to show you how to configure Linked Service to retrieve the password (keys) from Azure Key Vault and avoid storing that information in ADF code, JSON files.\n","date":"2020-02-12T07:00:49Z","image":"/uploads/2020/02/2020-02-11-Videos-Thumbnail.png","permalink":"/2020/02/adf-and-passwords-with-azure-key-vault-set-up-git/","title":"ADF and passwords with Azure Key Vault \u0026 set up GIT"},{"content":"Hello. We have a beautiful date today, haven\u0026rsquo;t we? And LWR posts are coming back after a long one-month break.\nPress Columnstore Indexes – part 130 (“Columnstore Indexes on Azure SQL DB”)\nNiko is gonna create the same CCI across many different version/plans of SQL engine.\nLate binding vs. Early binding in ADF Data Flows\nCheck the differences and when late binding scenario might be useful to use.\nOverview of load-balancing options in Azure\nGreat overview, particularly the decision-diagram.\nProcessing Geospatial Data at Scale With Databricks\nTake a look at how the Databricks Unified Data Analytics Platform can easily scale geospatial workloads.\nDifferent ways to update XML using XQuery in SQL Server\nSometimes still you need to do the stuff with XML.\nPolybase User Guide - Part 2: Unrestricted Storage the original Configuration\nMake a connection using Polybase as quickly as possible.\nUsing the Azure Data Factory switch activity\nWith example to run different size of a cluster as per the environment.\nStrange PowerShell Notebook Behavior When Comments Are Involved\nIt\u0026rsquo;s worth to remind how PS Notebook works in Azure Data Studio.\nA word from Microsoft’s CEO This is the decade for urgent action to address our world’s most pressing challenges. We know no one company can solve these issues alone, but here are my thoughts on how we must commit to do more in order for us to collectively achieve more. https://t.co/5GDjVu7mCU\n— Satya Nadella (@satyanadella) January 19, 2020 Video: Using Azure DevOps for Microsoft SQL Databases with SSDT https://www.youtube.com/watch?v=ObgY4XB0hHo\nSmile Corner ","date":"2020-02-02T23:32:01Z","image":"/uploads/2019/11/news-tyler-nix-tPz-uoz5RBU-unsplash-1200.jpg","permalink":"/2020/02/last-week-reading-2020-02-02/","title":"Last Week Reading (2020-02-02)"},{"content":" Introduction Uwe Ricken is working with IT systems since the 90’s. He found the way to the technology of Microsoft SQL Server with the assignment for the development of membership administration software for the American Chamber of Commerce in Germany. Afterwards, the software has been distributed to five additional European countries.\nThe primary passion for developments with Microsoft SQL Server expanded in 2007 with his engagement as a DBA for the Deutsche Bank AG. After 6 years of experiences in the European Operations Center as a DBA and over 14 years as a developer of complex database models he earned in May 2013 the “Microsoft Certified Master – SQL Server 2008” certification which “was” the highest technical certification from Microsoft.\nThe successful year 2013 has been finalized with the first MVP award for his outstanding support to the Microsoft SQL Server community in Germany and Europe. Uwe Ricken was the first MCM + MVP in Germany!\nUwe is blogging since 2010 about his daily experiences with SQL Server. His blog posts are in German language only to provide the German-speaking SQL community a deeper inside view into the technology. Uwe Ricken is a speaker on many international conferences and events and preferred topics for “Database Internals”, “Indexing” and “Development”.\nThis talk has taken place after SQL Saturday 2019 in Madrid, Spain on 28 September 2019 (Saturday).\nInterviewer: Kamil Nowinski (T).\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript KN: This is the next episode, and currently we\u0026rsquo;re sitting in Madrid for SQLSaturday and my next guest for this episode is my friend from Germany. And basically, I was planning to do this episode at least for a year, but I was planning to do this in Germany, because he is from there, but never happened, I was not here in the last period. But finally, I decided, so I must do it. I must do it finally to not skip this opportunity. So could you introduce yourself?\nUR: Oh yes, sure, so first of all \u0026ldquo;dzień dobry\u0026rdquo;! Dzień dobry, Kamil, dzień dobry, my friends in Poland. My name is Uwe Ricken, I\u0026rsquo;m from Germany, as Kamil has said. I\u0026rsquo;m 55 years old and I\u0026rsquo;m working with Microsoft SQL Server since 1998 I think, and I\u0026rsquo;m on international conferences since 2013. I love to work with SQL Server, it\u0026rsquo;s my daily job. The good thing is I love SQL Server, the bad thing is I only know SQL Server. No clue about anything else.\nKN: Oh, that\u0026rsquo;s probably not true. If you know SQL Server, so like a new version, about 2019, Big Data Clusters, so probably you\u0026rsquo;re starting learning new features.\nUR: Yeah, but big data cluster that\u0026rsquo;s not my business because most of my customers mostly have on-prem SQL Server running with 2016/2017. 2019 is not yet there. I think it will come in November, maybe. But I\u0026rsquo;m currently so busy that I didn\u0026rsquo;t have even one look into SQL 2019, although I know that there are so many pretty cool, good features but I was not able to play around with them. Business is, as I said, the main purpose currently.\nKN: Yeah, true. And you haven\u0026rsquo;t enclosed your age. I didn\u0026rsquo;t ask about it. (laugh)\nUR: Well, as you know, it\u0026rsquo;s quite important because many people think \u0026ldquo;Oh, look there\u0026rsquo;s this young guy\u0026rdquo;, and then I need to make clear that I\u0026rsquo;m 55 years old.\nKN: So where do you live?\nUR: I live in the area of Frankfurt am Main. That\u0026rsquo;s the best place to live there because I\u0026rsquo;m working international, I\u0026rsquo;m travelling a lot and therefore for me, it\u0026rsquo;s mandatory to have a big international airport in my area. I originally came from Ruhrgebiet, maybe you know it - coal. That\u0026rsquo;s more in the middle of Germany and I moved over in 1988 to Frankfurt to study law. And then my career started as an IT specialist.\nKN: To study law?\nUR: Yes, I needed the money. I was young and I saw a request for teaching people in IT. I said \u0026ldquo;OK, I\u0026rsquo;ve taught little kids then I could maybe teach adults too\u0026rdquo;. And so I started my career with the IT.\nKN: OK, so you started to like this, interesting. So what kind of take technology was valid at that time?\nUR: Oh my goodness, I think most of the\u0026hellip; Maybe you know DOS?\nKN: Yes, absolutely, I remember.\nUR: Disk Operating System [DOS]. That was my beginning, my very first beginning. And the machine I had was an x86 with 1 MB of RAM.\nKN: It was even before the 286, right?\nUR: Yes, it was the very first one, yes. And I started with 640 kB of memory. There was a DOS system. And I needed a computer because when I did my study at law, I, first of all, bought an electrical type machine, typing machine. But the problem there is, if you failed with something, you had to completely rewrite the complete page. So I decided to buy my very first computer then. It was in 1990/1991, because then before I printed it out, I could fix everything. That was the point.\nKN: And save a lot of time.\nUR: It saved lots of time, yeah. But I was very unprofessional with that. It was Word for DOS. I think that nobody knows it that something like this software was on the market at the time.\nKN: Only we dinosaurs remember those systems. Young people can\u0026rsquo;t even understand how it was possible that Google didn\u0026rsquo;t exist before.\nUR: Yes, it\u0026rsquo;s the same as with the mobile. I remember when I showed my little daughter a telephone with a\u0026hellip; how do you say, where you have to select the numbers with the circle (rotary dial). They were looking at this \u0026ldquo;my goodness, what is this?\u0026rdquo;. So yeah, we are definitely dinosaurs. The good thing was when I was doing the training, I did it for a teaching company and lots of these attendees, they liked my show, what I\u0026rsquo;m doing, when I was presenting. And they asked me to do a job in their prem, on their prem. So then I started my career as a developer. There was the very first time in 1993 as a developer for\u0026hellip; I think nobody knows this software. It was the Ashton-Tate Framework. Nobody knows it so far. But Ashton-Tate dBase?\nKN: Sounds familiar\u0026hellip;\nUR: It came out before Paradox. So dBase and Ashton-Tate Framework, you see, really old. Then I started my career with an American company, Motorola. I did the network there and that was my very first time when I started networking with Novell NetWare.\nKN: Oh yeah, I remember.\nUR: It\u0026rsquo;s on the market but it\u0026rsquo;s very, very tiny. At that time it was the biggest competitor for networking. And then came Microsoft with Windows.\nKN: My first serious job was at the company where they had Novell NetWare. But it was a German company in Poland.\nUR: And that was the very first time in my life I did a certification. The very first time I did a \u0026ldquo;Novell-certified blah, blah\u0026rdquo;, I don\u0026rsquo;t know what, it\u0026rsquo;s so long ago. But that was the very first time I got my first certificate in IT sciences.\nKN: You had to be very proud.\nUR: I was, I was at that time. Then it was going on and on. I was working with Microsoft Access, you know Microsoft Access? I think, most of the people know it. Then in 1998, I got a very, very big challenge. I had a customer, they wanted us to completely re-develop their membership software. And that was the very first time I started quite extensively with SQL Server. No idea about it, absolutely no idea. I did it from scratch. I said \u0026ldquo;oh my goodness, an index might be good here\u0026rdquo;, just for example. I didn\u0026rsquo;t know whether it is or not. But I was thinking about \u0026ldquo;well, when you search for it, maybe you use an index\u0026rdquo;. When I see today this software\u0026hellip; I have it on my computer. I facepalm. My goodness, I\u0026rsquo;m so happy about this customer and I\u0026rsquo;m so sorry about this customer because they paid me so much money for this crappy software.\nKN: But you finished the project. I think that is a goal for every project.\nUR: It was a very successful thing because we sold it in different countries. By the way, to Poland. The Polish chamber, they bought our software, and that was the very first time I was in contact with the Polish business. I loved it. It was in Warszawa (Warsaw), beautiful. So we had our software running in seven countries. I had a partner with me in the company, so we decided to go separate ways in 2007, like a divorce, right? Then I ran my own business again since 2007 and I didn\u0026rsquo;t have the manpower to support it anymore. I promised to my customer I will do. As long as you have the software, I would support you. And I was really, really happy when the last customer in 2012/13 said: \u0026ldquo;we\u0026rsquo;re going for new software\u0026rdquo;. I helped them to put the data into the new software. And now I\u0026rsquo;m doing what I want to do - SQL Server only.\nKN: OK, so you started describing how you started with your business. How was it at the beginning? How did you create it?\nUR: You mean in the 90s or the SQL Server business?\nKN: In the 90s, that was your first company, right?\nUR: To be honest, it was absolutely easy. Because IT was not so complicated as it is today. Just to give you an example. When I was working for an American company, we only had for 50-70 people just one server. We were running five services on it, we were running Windows because in Windows 3.11 you could put your profile on the server, mailing and print services. Everything on this one server. And, my goodness, the server was standing in my office. That was the 90s. I was aware of how professional IT must be the very first time, I would say, in 2001/2002, around that time. Then I saw the first time that they had a data centre for example. No one could access servers, all this kind of stuff. And then when I started working for banks, then I saw it on a very, very high, professional way. Segregation of duty, that means you do not have access to the Windows system when you\u0026rsquo;re a DBA, something like that. And I would love to see more companies working with this kind of segregation of duties.\nKN: Then you learn something new, what it should look like. Did you try to adopt this approach to every single company, every single customer of you?\nUR: I try to, but the big point is, to be honest, you don\u0026rsquo;t even have big, big companies. You have small companies, you have medium-sized companies and they don\u0026rsquo;t have the money for it. Just to give you an example, I have one company, 200 people working there, and they have just only three people working in IT. And these three people are backups for them, and they do everything. And for me, it is good, because they need experienced workers for SQL Server, but the problem is everybody has to do everything. And that is not a good way to work. But if you have big, big companies, corporates, they are strongly working on this segregation, which is very good.\nKN: These days there are fewer and fewer small companies on the market. But also I remember my first job was in a pretty small company where I was doing everything in IT. So from some perspective it\u0026rsquo;s good but from another, it\u0026rsquo;s bad, because you can\u0026rsquo;t focus on one thing, you cannot specialize yourself.\nUR: Well, it depends on what you want. So to be honest, for me, it is every single time quite annoying working for corporates for one simple reason: well, they pay me by the hour - this is okay for me, so the longer it takes for me to do a job, the better it is for me. But on the other hand, if you want to work something out, you want to fix something, and you can\u0026rsquo;t, this is quite annoying. And this is really disappointing. And this always happens in big companies, especially when working for banks. And just to give you an example, they hired me for an investigation of a SQL server. I saw the problem, I say \u0026ldquo;I made a document about the issues we found\u0026rdquo;, it said: this needs to be fixed, go through all this bullet list. And I said \u0026ldquo;OK, I can do it for you, it takes me 4 hours\u0026rdquo;. Do you know how long I was working there? Nearly a month. For 4 hours of work. And this is because of complicated processes.\nKN: You mean you were working for a month because they didn\u0026rsquo;t order the service from you?\nUR: No, they ordered it. I was working there, sitting there. The problem is I was waiting there. Because what you have to do in big companies is you have to create an RFC (Request For Change). On a production system, it is only allowed to do it on the weekend. You need to have the approval from the business, change management, implementer, all this kind of stuff. And maybe you know it, if they outsource services to India, Romania, wherever, the moment you outsource services. The people outside, they say \u0026ldquo;I don\u0026rsquo;t give a shit on it\u0026rdquo; because if it\u0026rsquo;s approved, I do, if not, I do not. Who cares. When we did the service in the bank, we were taking care that every single step has been approved. We were calling the people, saying \u0026ldquo;hey, I\u0026rsquo;m missing your approval, can you please click it, because I want to do it on this weekend, you\u0026rsquo;re waiting for this\u0026rdquo;. But if you outsource it, there is a big distance from culture and from distance and that makes it quite difficult. So that was for me easy working because one month I could do other stuff, but as I said, it was OK because I could do other stuff, on the other hand, it was very frustrating. And you will see very often in big companies.\nKN: Yes, I\u0026rsquo;ve seen. When I started working in London in the UK which was in 2015, I just started seeing where the problem is, what the processes really look like in big companies. So I never experienced something like this in Poland before. But it probably depends on what kind of company you\u0026rsquo;re working for.\nUR: Well, it is a highly professional thing what they are doing, but the problem is, at that moment I can understand the business manager when they say \u0026ldquo;IT is expensive\u0026rdquo;. For one simple reason: if you have lots of complicated processes, you involve more and more people into the process, and then it becomes expensive.\nKN: You want to be safe and this costs money.\nUR: All of them want to cover their asses. That\u0026rsquo;s it. Really, currently, I have one customer which I was working last week for and I explained the problem, I described the problem, I demoed the problem, it\u0026rsquo;s a small company, and I said: \u0026ldquo;let\u0026rsquo;s fix this and you will be happy\u0026rdquo;. Now they are thinking about 3-4 days now \u0026ldquo;shall we do, shall we not\u0026rdquo;. They are afraid of it. I say \u0026ldquo;come on, you hired me as an expert, why don\u0026rsquo;t you believe it?\u0026rdquo;.\nKN: If some students would propose those changes, might be OK, you can check this with someone else.\nUR: Yeah, but I\u0026rsquo;ve been on the market for 30 years, I know what I do. And to be honest, no risk no fun. I said, \u0026ldquo;come on, it\u0026rsquo;s easy, we\u0026rsquo;re not changing data, we only change the configuration, which can easily be reverted, don\u0026rsquo;t worry about it, I promise it will work\u0026rdquo;. They are a little bit afraid of it, so they don\u0026rsquo;t trust me.\nKN: Maybe they don\u0026rsquo;t trust themselves.\nUR: They are afraid of\u0026hellip; maybe they get some pressure from the business, from the management. I don\u0026rsquo;t care about it.\nKN: There are a lot of factors, yeah. I don\u0026rsquo;t care about it as well. But it\u0026rsquo;s hard that we can do something quickly, we know what is going on. We know what needs to be fixed and you need to wait because of the process, very long. So when exactly did you start working with SQL Server then?\nUR: I first touched SQL Server in 1998/1999, like that. But only for one simple reason: I saw the installation media said \u0026ldquo;SQL Server\u0026rdquo;. What is it? I installed it. Internet was not available at that time. I read a book about, it said: \u0026ldquo;SQL Server is a database system\u0026rdquo;. I said, \u0026ldquo;oh my goodness\u0026rdquo;. And here comes the big point. I was coming from Microsoft Access. Access was OK, I loved it, but here comes the big, big point. At that moment, when I discovered SQL Server, it was the version 4.21, so very, very old. And that is a big, big problem I think that many developers today have too: I expected if I put my data on a SQL Server, everything will run faster. I completely failed. I thought that now everything would be fine and easy and incredibly fast. I failed completely. But that was the very first time. And I thought by myself, that was the second misunderstanding, I was thinking about SQL Server as a development environment. I was searching for \u0026ldquo;how can I develop GUIs or something like that\u0026rdquo;. Very funny, today I can laugh about it but at that time it was incredible. And it was then I got my first\u0026hellip; 4.2 was in 1996/1997, not 2000. And in 1998 I got my first engagement for the development of a SQL Server-based application. And from 1998, to be honest I learned very, very much about development with SQL Server that took me 10 years. But I didn\u0026rsquo;t have any clue about security, installation, performance, nothing.\nKN: At the beginning, you just learn how to create the system, the application.\nUR: I was focused on development. But SQL Server is way more than development. You have two separate parties: development and administration. So in 2008, I started at Deutsche Bank in Germany. That was the very first time I started as a DBA.\nKN: 2008, really?!\nUR: Yes. And the funny thing is they hired me although I said I don\u0026rsquo;t know anything about DBA work. But they were impressed. In the interview, they told me about mirroring. I said, \u0026ldquo;well, mirroring, it\u0026rsquo;s easy, I did it\u0026rdquo;. The point is in SQL Server 6 mirroring did mean that you had a database on the same server, not mirroring between two servers, something like that. But the key point was, he heard that I understand mirroring and they said: \u0026ldquo;OK, you\u0026rsquo;re my guy, you can come\u0026rdquo;. So since 2008, I got the professional way to work with SQL Server. The very first two years was for me learning, learning, learning from the DBA aspect. But the good thing was my three colleagues, they were only familiar with the GUI. They could do every single job but they always needed the GUI. And the problem is that it\u0026rsquo;s not effective enough, especially when you have repeatable things. So what I did then was, you know in the GUI you click, click, click and then you say \u0026ldquo;scripted\u0026rdquo;. And this was the way I learned to administer SQL Server by T-SQL. And what I did that was I was developing scripts for the administration of SQL Server for our team. And that was from 2008 to 2010. In 2010 they started outsourcing the services and those were the two worst years I would say because there was nothing more to do, only a few things, for that customer. But 2010 was I would say the big point where I changed completely my SQL Server experience because one of my colleagues was telling me about the “SQL Server Master” program. And he was coming to me and said: \u0026ldquo;Uwe, I heard about SQL master, this is the highest level of technology you can achieve in Microsoft certification, and now I can do it in an easy way, I don\u0026rsquo;t have to go to Redmond, I will learn it\u0026rdquo;. And I said \u0026ldquo;what is it?\u0026rdquo;, and he was \u0026ldquo;yeah, it\u0026rsquo;s the best thing you could ever do, you learn much and you are a well-renowned engineer\u0026rdquo;, all this kind of stuff. I said, \u0026ldquo;OK if you do it, I do it too\u0026rdquo;. And that was a competition. That was good for us both. So what was the first step? I did all my recertification, so the MSCE, MCSA, because you needed to have four certifications at least, and then I started watching the videos from [Kimberly [Tripp]](https://twitter.com/KimberlyLTripp), from Paul Randal, from [Brent [Ozar]](https://twitter.com/BrentO). It took me two years to prepare for the master. And then happy, happy, happy, I got it in 2013. So it took me two and a half years. But you know, what impressed me most was when I started in 2010, I was thinking I know everything about SQL Server. What a big, big, big mistake. And when I started learning that, I saw how little I knew at that time, how incredibly little I knew at that time. Today I\u0026rsquo;m happy that I have it. The MCM program is dead, but the knowledge you can transfer to every new version, it\u0026rsquo;s always the same, and that\u0026rsquo;s a good thing. The basics are the same.\nKN: Everyone has the same I think. I don\u0026rsquo;t remember the name of this chart that\u0026rsquo;s showing you how much experience you have, how long you\u0026rsquo;ve been working in the markets and how much you think you are experienced and then when you learn more, it\u0026rsquo;s going down.\nUR: Well, Microsoft itself recommends 3 to 5 years experience. To be honest it\u0026rsquo;s too little. To pass it, it\u0026rsquo;s too little. But you could do the boot camp, you could at that time do the boot camp in Redmond, but the problem is it\u0026rsquo;s way too expensive. It was $20,000 I think, I don\u0026rsquo;t know. But flight costs, accommodation, no income for three weeks. So that meant for me, I was calculating, it was about forty thousand euros. I said \u0026ldquo;no, it\u0026rsquo;s not worth it\u0026rdquo;, at that point, at that time. But then, when they changed the program, they changed it and said: \u0026ldquo;OK, you can prepare by yourself, you don\u0026rsquo;t have to go to the boot camp, get your own pace, here is the stuff, the only thing you have to learn\u0026rdquo;. I did it, took me two years and that\u0026rsquo;s it.\nKN: So before it was like you had to do the course at the place for these three weeks and then finish with the exam.\nUR: Yes, but it was required to go to the boot camp. But now, as you know, it\u0026rsquo;s dead. Don\u0026rsquo;t talk about it anymore, it\u0026rsquo;s done.\nKN: But also there\u0026rsquo;s the MVP program and you\u0026rsquo;re still an MVP, so you\u0026rsquo;ve been an MVP since 2013.\nUR: April 2013. It was at the same time when I had my master. When I received the email from Microsoft, to be honest, I was pretty proud of it. Because it showed me that my efforts I\u0026rsquo;m doing for the community have been recognized by Microsoft. And I love to be an MVP, that\u0026rsquo;s a great community, you know it by yourself, so I shouldn\u0026rsquo;t explain it to you, but the MVP for me it means\u0026hellip; well, it\u0026rsquo;s a giving back from the community to the people who engage for the community. For example, now we are here both sitting in Madrid. But we are doing everything on our own cost. I don\u0026rsquo;t think that many people know that, but every single SQLSaturday you\u0026rsquo;re on, it\u0026rsquo;s important to know for the audience, every speaker is doing it for free. The cost is completely on their side.\nKN: There\u0026rsquo;s no remuneration for that.\nUR: Therefore I would say it\u0026rsquo;s really cool that Microsoft recognized it. I mean not this only but that\u0026rsquo;s one part of the MVP.\nKN: A lot of activities for the community, technology, around this.\nUR: And I need to say that maybe a few people think \u0026ldquo;come on, MVP, it\u0026rsquo;s easy to achieve\u0026rdquo;. No, it isn\u0026rsquo;t. Because you have to engage for the community by heart. Not only say \u0026ldquo;OK, let\u0026rsquo;s go to the forum, make 10,000 points and maybe then I achieve the MVP\u0026rdquo;. That will not work. Not yet.\nKN: Even if you achieve that like this, maybe by mistake or by accident, maybe it will be not extended for the next years. Because we need to mention that the MVP program is not given to you for the whole life, it\u0026rsquo;s being extended every year, so you need to continue what you\u0026rsquo;re doing.\nUR: You see how many fabulous people they have thrown out of the MVP program. People I would never have expected. [William [Durkin]](https://twitter.com/sql_williamd), that was really a big surprise. He\u0026rsquo;s running Data Grillen and I was wondering why they kicked these people out. Or maybe you remember, that was two or three years, Paul White. Mr Paul White, Mr SQL engine, he\u0026rsquo;s brilliant in his stuff. He\u0026rsquo;s pushing his work into the community, and I love his work because he\u0026rsquo;s so exceptional. There are lots of them. I don\u0026rsquo;t want to name them all but I would say, people who are MVP, they are brilliant people, they love to work or do community work. Maybe not everyone but I would say over 99%. And I have to be honest, the MVP for me, for my business it was boosting my business too. I don\u0026rsquo;t have to hide behind it. That\u0026rsquo;s a true fact. If you ask a customer \u0026ldquo;do you know MCM\u0026rdquo;, or \u0026ldquo;do you know MVP\u0026rdquo;, you get over 90 per cent \u0026ldquo;well, I know MVP, yeah, they are experts\u0026rdquo;. MCM, mostly in Germany, they didn\u0026rsquo;t know what it is. No chance, they didn\u0026rsquo;t know. But it\u0026rsquo;s OK because let\u0026rsquo;s go ahead with the MCM, it\u0026rsquo;s over since 2013.\nKN: So in terms of MVP, are you still blogging?\nUR: Sure, but unfortunately way too little. I have so many topics. You need to know, in my calendar, in my Outlook I have an activity list what blog posts I want to write. What do you think, how many topics I have there? Over 30. But the problem is, I don\u0026rsquo;t have the time currently to do this. I want to do this, I really want to do it because they are extraordinary things in it. For example, the last blog post was about bacpac problem, with bacpac fights when you deploy your database to Azure database and suddenly your application is running mad. That can happen and the reason is in the process of bacpac\u0026hellip; but this was the last thing I was writing about. But I have lots of other things to write because I love writing about the SQL engine, it\u0026rsquo;s an interesting thing.\nKN: I saw it on the MVP group (Distribution List).\nUR: Yeah, but that is one of the, I would say, bad points is I only write in the German language.\nKN: Yeah, that was my next question: why only in German?\nUR: That\u0026rsquo;s very simple to explain. Two things: first, I\u0026rsquo;ve never seen a good German blog post or an intensive German blog post, let\u0026rsquo;s put it this word. Which means if I would do it in English, there are so many, many, many, hundreds of brilliant bloggers which are doing SQL stuff in English. And they are doing a fantastic, an absolutely fantastic job. So my decision was directly in 2012 when I started blogging, I said: \u0026ldquo;no, I do it only in German\u0026rdquo;. Because the feedback of the German community was \u0026ldquo;oh my goodness, we don\u0026rsquo;t find blogs in german language, we would love to have it in German\u0026rdquo; and that\u0026rsquo;s how the idea was born that I would blog in German. What I do is sometimes I\u0026rsquo;m blogging for Redgate. Then I take my blog post, rewrite it in English. And now I have contact from Israel. He was contacting me and asked me whether he could translate my blog. I\u0026rsquo;m feeling really honoured. He asked me if he could translate my blog posts to English, that he could post it on his website. Do what you want, it\u0026rsquo;s for the community, give it for free. I was thinking about doing it in two languages. But first is it\u0026rsquo;s complicated and second is then I need double the time.\nKN: And that stops you from doing that. I had the same problem when I started my blog. Thinking \u0026ldquo;should I write a blog post and keep it in English because I\u0026rsquo;m living and working in the UK now, or should I do it in Polish or both?\u0026rdquo;, so I thought like \u0026ldquo;both would too complicated, too time-consuming, it would be stopping me from doing that instead of starting doing this\u0026rdquo;.\nUR: Another point with blogging is for me it is a marketing tool for me. I\u0026rsquo;m not so professional as for example Brent Ozar, he\u0026rsquo;s brilliant in marketing. He\u0026rsquo;s a master of marketing, I would say. But I\u0026rsquo;m not. But I would say I get at least once a week from my contact form a request if I could do some services. And they mostly are landing on my blog posts because they are searching for it. All my customers are coming from Austria, Switzerland, Germany. I have two customers in Singapore and one in Hong Kong. The most far customers I have. OK, I speak with them in English but you see, most of my customers are coming from the German area so why shouldn\u0026rsquo;t I? This is my market, you know? And as I said there are brilliant English posts that why should I get in competition with these guys.\nKN: You also travel a lot. Do you like travelling now? You can confess.\nUR: So let me separate two different kinds of travelling. First is business travelling. So under the week from Monday to Friday, I travel really a lot. So just for your understanding: I travel by car, per year, around 60,000 kilometres and by plane (I don\u0026rsquo;t use trains, they are not reliable in Germany) I would say about 50-60 business flights, only in Germany. 60 flights a year. I have around 150 nights per year in a hotel for business. So that means I\u0026rsquo;m travelling around Germany, wherever someone is requesting a job I go there. And then it comes to the weekends - SQLSaturdays. The highest amount of SQLSaturdays I had I think in 2015/2016, it was around 20. That means 20 weekends, out of 52, was travelling around for conferences because it\u0026rsquo;s mandatory for me. I love to travel around for conferences. I love to see the speakers. For me, it\u0026rsquo;s a little bit of a family. Yesterday you saw it when we were in the restaurants, I love to see you all again. I love it.\nKN: I’m feeling the same, it\u0026rsquo;s like an addiction.\nUR: It is! So that takes me another three days. And here comes the point, when I travel for conferences, the good thing is I see different countries, I get more open-minded at the same time.\nKN: You can also relax a little bit and see some places, taste new food, meet people.\nUR: Yes, and so in Europe, I would say I was nearly at every SQLSaturday since my career as a speaker. Last year the farthest I travelled was to South Africa. Beautiful, I love it.\nKN: That was my other question, you\u0026rsquo;re just reading my mind!\nUR: And next year, that\u0026rsquo;s really a highlight. I have to keep it safe. Next year I\u0026rsquo;ll be in New Zealand for Difinity 2020 in Auckland, and this will be the longest flight in my lifetime. It is 6.5 hours to Dubai and then from Dubai non-stop to Auckland 16 hours. Let\u0026rsquo;s see how it works.\nKN: Are you preparing for that mentally?\nUR: Well, I have an e-book reader.\nKN: Also, I was planning to be there, in Auckland, I was talking about this with Reza and Leila. Maybe next year, maybe not in 2020, but in 2021. We\u0026rsquo;ll see. I need to plan in advance. I just changed jobs recently so it was very hard to make that happen. New projects, new employer\u0026hellip;\nUR: What I tried to do is then when I\u0026rsquo;m in Auckland, I do the Difinity, then fly over to Wellington in the south, do the SQLSaturday, fly back to Auckland and then fly back home. And the big, big challenge next year, to be honest, this is really a big challenge, the Difinity is in February, so I come back from Difinity, 12 hours time zone difference, come back to Frankfurt, one week later I fly to Canada for the SQLSaturday, if it happens SQLSaturday in Victoria, and go to the MVP Summit. So that means 9 hours. Back. I will see how it works. I\u0026rsquo;m not sure if I could manage it. But if not, then I know my limits.\nKN: Yeah, so I think we should sit for 5-10 minutes, summarize your experience during the MVP Summit.\nUR: Yeah, why not. As long as there\u0026rsquo;s no Information about internal data we can talk about it.\nKN: So let\u0026rsquo;s talk about the SQLSaturday in Johannesburg, because I\u0026rsquo;ve seen that you started flying there like a year ago or two years ago.\nUR: I started with\u0026hellip; let me think about it. 2016 I started. That was the very first time. So, Michael Johnson, I was in contact with him, I submitted for SQLSaturday in Joburg because for one simple reason, I was never in my lifetime in South Africa. So I was asking my wife whether she would join me, and we made a big plan, we said \u0026ldquo;OK, let\u0026rsquo;s go one week before, go to a game lodge, see the lions, all this kind of stuff, elephants, rhinos and so on. Then we went to Joburg only, in 2016. We only did the conference in Joburg and then we flew back to Frankfurt. So it was one week, it was easy because of the same time zone, everything fine. But Joburg was\u0026hellip; how do I say, we didn\u0026rsquo;t see anything from Joburg for one simple reason: people recommended not to go out of the hotel. It\u0026rsquo;s so dangerous in Joburg.\nKN: Yeah, exactly, I wanted to ask if it\u0026rsquo;s safe there.\nUR: I do not really know. So what we did was, and that was a great, great, great, really, thank you to Michael, that was a great hospitality, they picked me up from the hotel, brought me to the speaker dinner and back, and to the venue, I did a pre-con there two times, and on the Saturday do the SQLSaturday and back to the hotel. Really, absolutely brilliant hospitality. So to be honest, I didn\u0026rsquo;t see very much of downtown. I only saw it once when we had the transfer to the airport because we took an Uber from the hotel to the airport and the highway was blocked, so he decided to drive through downtown. I didn\u0026rsquo;t feel very comfortable, I have to admit. So I think the recommendation from people who are living there is absolutely valid when they say \u0026ldquo;please, try not to leave the hotel\u0026rdquo;. But I think if you take an Uber, you\u0026rsquo;re safe. But we didn\u0026rsquo;t do this. Why should I take a risk? If someone warned me and say \u0026ldquo;no, better not\u0026rdquo;, then why should I do this? So that was Joburg. We had a lovely hotel, I have to say. Beautiful hotel. The next year we skipped it, and then I asked my wife again, said \u0026ldquo;what are you thinking about it, shall we go again to South Africa?\u0026rdquo;, she said \u0026ldquo;oh my goodness, yeah, yeah, let\u0026rsquo;s do it!\u0026rdquo;, I said, \u0026ldquo;OK, kochanie (the polish word for sweetheart), we do\u0026rdquo;. And so I booked for Joburg, and not only Joburg but one week later it was in Cape Town. So what the guys in South Africa are doing is absolutely brilliant. They have three SQLSaturdays in a row. So that makes it easy for speakers. Week by week by week. So we went to Joburg, I did the pre-con and the SQLSaturday there, then we flew over to Cape Town, do the pre-con and the SQLSaturday there, and then we flew back to Joburg and home. Next year, if they take me I don\u0026rsquo;t know, but then we will do Joburg, Cape Town, Durban. All three. And then we go back to Frankfurt. And I really love to fly long-distance flights, because we take business class, it\u0026rsquo;s easy, you can sleep and we have a night flight basically, we have enough place, we can read, we can do whatever you want to do. So that was really my\u0026hellip; was it my longest flight? No, I was in India, the Data Platform Geek Summit, and from India, I went over to Singapore for a week and then back to Germany. So flying long-distance, going to SQLSaturdays all around the world is fantastic. The only area which I currently missed totally was the US. I need to see, let\u0026rsquo;s see, 3 SQLSaturdays in a row, I have to pick them up, and then I have to arrange a flight, accommodation, all this kind of stuff. But the US, well\u0026hellip; I am addicted to Asia. I\u0026rsquo;m completely addicted to Asia. It\u0026rsquo;s beautiful, the people are so beautiful, they are so kind, really, they are lovely. And every once a year I\u0026rsquo;m in Singapore, every year, I love to be in Singapore. I was last weekend there for Formula 1 and I\u0026rsquo;m absolutely addicted to it. I was in Vietnam, I was in Indonesia, I was in Thailand. Unfortunately, there is no SQLSaturday there because I would immediately do it. Bangkok, what a beautiful city. It\u0026rsquo;s amazing. They have a New Year\u0026rsquo;s celebration. So I strongly recommend it. It\u0026rsquo;s beautiful. That\u0026rsquo;s the reason why I go to SQLSaturdays. It\u0026rsquo;s on the weekend, I can see different nations, different cities, different people. And I have some favourites, to be honest. I\u0026rsquo;m addicted to Poland. That\u0026rsquo;s not only for the interview. I\u0026rsquo;m four times a year in Poland. In Krakow, in Wrocław (SQLDay), in Warszawa (Warsaw) and Opole but Opole is more of a private thing. My wife is from Poland.\nKN: We\u0026rsquo;ll come to this. So the next question will be also about the conferences, a quick one. Do you remember what was the biggest conference that you have taken part of as a speaker?\nUR: Well, to be honest, that was the PASS Summit in 2015 I think. And the second biggest was the Data Platform Geek Summit in India. Perfect. It\u0026rsquo;s amazing, the more people are coming to my session, the more I\u0026rsquo;m completely excited and I get out of myself. So at the PASS Summit, I was not so lucky with the session for one simple reason: it was on a Friday because many people went home Friday, it was after lunch. But I had 130-140 people in my session. And it was in a big ballroom. You know the PASS Summit and you know the ballrooms, right?\nAnd I remember it as it was yesterday. When I first saw\u0026hellip; You know me as a speaker on this big, big platform, waiting for the audience coming in, and you know that there could be thousands of people in, and then you saw 100-130, I was a little bit disappointed. But however, that\u0026rsquo;s my intention: whether there is one sitting there or 1000, I always try to give my very best. The people should have fun when they are on my session. I love it when they laugh. That makes it so easy. And to be honest, if you laugh about something, you keep it in your mind. You say \u0026ldquo;oh my goodness, yeah, that was his joke about this, ah, indexing\u0026hellip;\u0026rdquo;. That\u0026rsquo;s how I do my sessions. And Bangalore was for me a very exciting thing because the greatest thing from DPG was the people are fascinated about learning. And they came into my session and the organizers warned me at my pre-con, said: \u0026ldquo;go straight through your topics, do not answer questions\u0026rdquo;. They were coming in my session before, introduced me and said: \u0026ldquo;please keep in mind this is a pre-con, no asking questions, and we will go through the stuff soon\u0026rdquo;. Oh my goodness, what am I doing here? And I allowed to ask questions. That was my fault. Because they wanted to learn and ask this and this, and this. I didn\u0026rsquo;t understand the mentality at that point, but it was brilliant, absolutely brilliant. Good hospitality, beautiful.\nKN: Another part of the conversation: what do you think is the nearest future of the data platform technology?\nUR: That\u0026rsquo;s a difficult question.\nKN: Cause everything is changing very quickly, extending, even SQL Server is expanding a lot these days.\nUR: Absolutely. When I was starting with SQL Server, we only had the platform, the engine, nothing else. OK, a little bit of SSIS, that\u0026rsquo;s it. And then came Analysis Services, Reporting Services, and you saw that the SQL suite was growing up and up. And see what\u0026rsquo;s happened now. You have lots of different technologies\u0026hellip;\nKN: Big data clusters, satellites with R, Python, Java\u0026hellip;\nUR: And this is something for me as a dinosaur for databases, which I, to be honest, completely dismiss. I don\u0026rsquo;t want to have it in a database because from my point of view, in relation database, that\u0026rsquo;s my point of view. When they started with XML, I said I was puking. I said, \u0026ldquo;my goodness, what are they doing there\u0026rdquo;. That was because of the developers. I know that the data-based platform should change, have more of a focus on the development but not only on the let\u0026rsquo;s say DBAs for example. Then cloud was up and coming, and I would definitely see that the future is in cloud computing but the problem I think it\u0026rsquo;s in every country is that the current management is not so\u0026hellip; They are concerned about cloud computing. If the new management comes into the positions, then it will speed up. In Germany, we have very big concerns about data security. This is the biggest point where people say \u0026ldquo;no, I don\u0026rsquo;t want to go into the cloud\u0026rdquo;. And I would say on-prem would ever be when I retire I would say there would be on-prem software, but artificial intelligence, KI, all this kind of stuff, that will go more and more into the focus. And what I would say is that the model, the database models, they will rapidly change. So if you\u0026rsquo;re talking today about relational databases, I would say the big data kind of stuff that could not be handled by relational\u0026hellip; So that\u0026rsquo;s the reason why they always change. NoSQL. I never got in touch with NoSQL, I don\u0026rsquo;t have any experience with that. For two reasons: first of all, I\u0026rsquo;m not so interested in really, and the second is I didn\u0026rsquo;t get a project with that. To me as a freelancer, you know it, I cannot sit around for one week and say \u0026ldquo;oh my goodness yeah, I love this new kind of stuff\u0026rdquo;.\nKN: And then you will not try it in practice, it doesn\u0026rsquo;t make sense.\nUR: And something that is really annoying, it\u0026rsquo;s not by Microsoft, different companies are doing that, they are pushing a feature, everybody says \u0026ldquo;oh yeah, my goodness, what a cool feature\u0026rdquo;. Next year say \u0026ldquo;sorry, we stopped it\u0026rdquo;. And what did you learn? This is your time, your effort for that, and then they say \u0026ldquo;don\u0026rsquo;t give a shit on it\u0026rdquo;.\nKN: I think the reason for this is the pace of how everything is changing. How quickly they want to adopt the technology.\nUR: Yeah, that\u0026rsquo;s the point. For me the period between new versions, from my point of view, it\u0026rsquo;s way too fast. For on-prem by the way, sure. I mean now dedicated SQL Server for example. It\u0026rsquo;s way too fast. I know there\u0026rsquo;s new technology, they have new features. And from my point of view, it is so that many cool features they are implementing is because of some stupidity of developers. No, that\u0026rsquo;s wrong because then I would blame developers. That\u0026rsquo;s not correct. Let me say it in other words. I know the problem of developers. Developers can make great GUIs, can make perfect services, but when we think about the data, for them it\u0026rsquo;s only \u0026ldquo;come on, I store it in a database\u0026rdquo;. That\u0026rsquo;s what it is for. Then I have to apologize for my stupid\u0026hellip;\nKN: From that perspective, those positions should be separated, you know front-end developers and back-end developers. Whatever it will be like, service or database. Or a database developer.\nUR: You won\u0026rsquo;t see them. In medium companies, they don\u0026rsquo;t even have it. They have developers who are doing everything. And that\u0026rsquo;s the reason why I can live from it.\nKN: But then we have work.\nUR: And that\u0026rsquo;s what I love about my work. You come in, people are concerned, \u0026ldquo;well is it the right man for doing this job\u0026rdquo;, and after let\u0026rsquo;s say 3-4 days you get out and you are the hero of the company. They say \u0026ldquo;ah, you saved our life, great, wonderful, come back whenever you want, you get free cookies!\u0026rdquo;.\nKN: True. OK, let\u0026rsquo;s leave the Azure things. Let\u0026rsquo;s go a little bit into your private life.\nUR: Oh, yes!\nKN: A simple one at the beginning: what is your hobby?\nUR: What is my hobby? Diving. We do it with the whole family. Unfortunately, in three weeks we wanted to go for diving, and that\u0026rsquo;s my annual holiday, but unfortunately, maybe you heard about it, the Thomas Cook bankruptcy, I\u0026rsquo;m a victim of it, so we can\u0026rsquo;t do this year. But diving is the hobby of the whole family. And another hobby is my private car from Italy. It\u0026rsquo;s a Ferrari, it\u0026rsquo;s a 30-year-old Ferrari. I\u0026rsquo;ve owned it for over 20 years now. 30 years now, it\u0026rsquo;s a historical car. That\u0026rsquo;s good, it\u0026rsquo;s cheaper for taxes. And my third hobby is Formula 1.\nKN: Driving or watching? [laughs]\nUR: Oh, come on. No, no, no. I\u0026rsquo;m looking at maybe visiting it, but to be honest I would say since five years, if I see it on TV, it\u0026rsquo;s okay. Let\u0026rsquo;s say if Ferrari wins, very great as they did in Singapore. But the focus is going now to different other things. I like to walk in forests, beautiful, have some calm, you know? Because my life is currently running so quickly. Every single day another customer, every single day another city. Do you know the situation? I\u0026rsquo;ve had it three times now. You go into your hotel room, take your keycard and the door won\u0026rsquo;t open. Then you get down and said \u0026ldquo;hey, come on, my door doesn\u0026rsquo;t open\u0026rdquo; and then they tell you \u0026ldquo;sorry, you were in the wrong door\u0026rdquo;. And you say \u0026ldquo;my goodness, it was yesterday where I was in a different city\u0026rdquo;. And I have these situations. And that seems that is for me came just really clear the point where I say \u0026ldquo;calm down, stop, not so fast, try to focus more on the family\u0026rdquo;. And that\u0026rsquo;s the reason why we love to go diving. Diving you need to be very concentrated, if you think about other things, it could be quite dangerous because if you\u0026rsquo;re not aware of what\u0026rsquo;s going on with you, it could be quite dangerous when you\u0026rsquo;re 30 meters down and you have some problems, you cannot go up immediately up to surface, it doesn\u0026rsquo;t work, it could kill you.\nKN: I tried a few years ago and yeah I understand the feeling. So it was only once but yeah, I\u0026rsquo;m not sure if I\u0026hellip; probably in the future I will repeat it but I was a little bit scared but I was repeating to myself \u0026ldquo;calm down, calm down, you need to focus on what you\u0026rsquo;re doing, breathe slowly, don\u0026rsquo;t panic\u0026rdquo;. Because this is the worst thing you can do under the water.\nUR: And that\u0026rsquo;s why I love diving. So when you go to let\u0026rsquo;s say the Maldives, to Egypt or something, you have to dive with the buddy. And my buddy is my wife. So you know, we both go together and we can completely\u0026hellip; I\u0026rsquo;m feeling quite safe when she is near to me and the opposite. So my daughter\u0026rsquo;s, they are the buddies for each other and then we go with four people we\u0026rsquo;re going diving. Really beautiful. And you know what the cool thing is when you sit at the evening at the bar, with a cocktail and you\u0026rsquo;re talking about what you see. I love it. That\u0026rsquo;s the way of family life I absolutely love.\nKN: It\u0026rsquo;s great that you have something in common in your family. Like going on holiday and doing the same thing. Then you can share the experience.\nUR: That is a great, great benefit we have. My oldest daughter she\u0026rsquo;s living 300 km away from us. The youngest one is living very near to us, but I always love to have my family around me. So whenever we have the opportunity for a holiday together in Italy, we were this year in Italy, or in November we go together to the Baltic Sea and we meet there, it\u0026rsquo;s a big family meeting. And to be honest, these are absolute highlights of the year when I see my family. I could not express how important it is for me to have a family around me. That gives me the power to do my daily job. I\u0026rsquo;m doing it for my family. I want to have them comfortable, I want to have them around me, and I want to have lucky people around me. When I see them for let\u0026rsquo;s say 3-4 days, we spread out to our locations, I\u0026rsquo;m absolutely happy. That gives me the speed for the next 3-4 months. That\u0026rsquo;s what I love.\nHow many minutes are we currently doing? An hour, that\u0026rsquo;s what you said, yeah.\nKN: Normally it takes between 30-60 minutes.\nUR: But I\u0026rsquo;m talking too much.\nKN: No, not too much, I knew that because I knew that you liked talking and that\u0026rsquo;s fine.\nUR: But you know, I\u0026rsquo;m 55 years old, I\u0026rsquo;ve seen lots of things in my life.\nKN: Exactly, so you know, this podcast is, that was the original idea, it\u0026rsquo;s not talking about technical things that you can learn during the SQLSaturday every session. That was not the point. The point was to share the experience from many areas, to share your approach to your private life, what you do, how you can cope with some things. This kind of things that people don\u0026rsquo;t know about us.\nUR: We are sitting here really relaxed.\nKN: Exactly! So,… a holiday. So you mentioned that your flight has been cancelled. And where were you planning to go for the holiday this year?\nUR: We wanted to go to the Maldives. We go every year to the Maldives for diving. We learned it there in 2012 and we are feeling very safe with the team of the dive school there. We know them and this is like coming home, you know? This year we cannot make it but what we will do now, unfortunately, they took my money, but we didn\u0026rsquo;t get anything for that, so we are now doing it this way: we booked in Egypt. Because I need to have two weeks of holiday. It\u0026rsquo;s important for me because I see that the batteries are low. I feel it. And I need to get a little bit of distance for two weeks, and therefore we decided to hopefully go to Egypt.\nKN: When will it be?\nUR: In three weeks, for two weeks. But you know, the problem is so many people cannot go to the holidays and now you see the prices increasing. They are looking for other things.\nKN: Yeah, that\u0026rsquo;s the bad thing. OK, so we know a little bit how the work-life balance works for you because you talked about it.\nUR: Yeah, I don\u0026rsquo;t have one.\nKN: I think you have!\nUR: But it\u0026rsquo;s too little.\nKN: You mean 2 weeks of holidays is too little?\nUR: Yeah, I have too little holiday. Because sometimes I hear my daughter saying \u0026ldquo;well, daddy, you\u0026rsquo;re going on the weekend to your conferences\u0026rdquo;. OK, yeah, it seems to be life balance because it\u0026rsquo;s easy, but it isn\u0026rsquo;t. It\u0026rsquo;s travelling, you\u0026rsquo;re not at home, and also I\u0026rsquo;m travelling very much by aeroplane, it is always stressful for me at the security. I hate it. And waiting for flights and boarding and looking for a place where it could drop your luggage. All this kind of stuff, this is stress. And travelling around means that you\u0026rsquo;re for three days\u0026hellip; just to give you an example: Madrid - yesterday we started our travelling at 10 (am), we\u0026rsquo;ve been in the hotel at 5 (pm), so five [seven] hours of travelling. When we go back, we start here, so the flight would go on 3 (pm), so that means we start here at 12 (pm), to the airport, and I think we\u0026rsquo;ll be home at 7 pm. Many hours. And all of us are doing it. Not only me. That\u0026rsquo;s all of you, all those people are doing this.\nKN: I was here in Madrid at 9/10 am, but I had to wake up at 4 am!\nUR: You see! So when I go to Hamburg for example, when I go for a business trip to Hamburg, I wake up at 4 am, because I take the first plane at 6:10 to Hamburg that I\u0026rsquo;m at the office at 8 am.\nKN: And then you are just tired.\nUR: I\u0026rsquo;m tired, and I feel it. I feel it now too. Also, I slept really well after the speaker dinner yesterday. I\u0026rsquo;m feeling a little bit tired. I know that it\u0026rsquo;s over when I start my session, but I feel that the batteries are going down. So two weeks of relaxing. Don\u0026rsquo;t think about SQL Server, anything, don\u0026rsquo;t think about business.\nKN: Don\u0026rsquo;t even touch the computer.\nUR: Yeah, that is unfortunately for me not possible. For one simple reason: first, I have a company, but that\u0026rsquo;s not the point. You know what I absolutely hate is when you come back into the office, open your laptop and you have 300-400 emails. It\u0026rsquo;s a nightmare for me. This is something I try to avoid, so what I\u0026rsquo;m doing is every evening I open the laptop, look through the emails. But I set a limit. I said 30 minutes, nothing more.\nKN: Because I wanted to ask you if you think it\u0026rsquo;s worth checking emails every evening instead of sitting for one day and checking everything, replying to the emails.\nUR: It depends. Let me tell you a story about it because that gives you a good impression that you shouldn\u0026rsquo;t do this. We were in 2014 I think, we\u0026rsquo;ve made a diving holiday in the Philippines. And the first week - perfect. No emails, didn\u0026rsquo;t touch the computer, anything. But on my mobile, I saw the emails. But what I was doing was \u0026ldquo;OK, not interesting, not interesting\u0026rdquo;. But then I got an alert from a company, from a corporate I was working for saying \u0026ldquo;please, can you help?\u0026rdquo;. But you need to know that the Philippines is 7 hours ahead of you, so what did I do? As a stupid guy, I said: \u0026ldquo;yes, sir\u0026rdquo;. I\u0026rsquo;m like Starship Enterprise: \u0026ldquo;Scotty, we have a problem, can you fix it?\u0026rdquo;, \u0026ldquo;yes, sir, how much time do I have?\u0026rdquo;, \u0026ldquo;Scotty, you have 6 hours\u0026rdquo;. Scotty then: \u0026ldquo;OK, captain, I\u0026rsquo;ll do it in 3\u0026rdquo;. Like this. That\u0026rsquo;s me, in the same way. So then we started emailing, I was going on the server. What shall I say, it took me the whole night. It was day time in Germany. That was an incident on the production system. It took me the whole night to fix the problem, the whole night. And what shall I say? The first week I was down. I didn\u0026rsquo;t think about anything of business. But after this night the rest of the holiday didn\u0026rsquo;t exist because I was always thinking \u0026ldquo;did you do everything right, can they work now, what will happen if they get in contact with you again, what else can you do?\u0026rdquo;. Blah, blah, blah, that wasn\u0026rsquo;t worth it, you know? I should normally have said \u0026ldquo;go and check it alone, I\u0026rsquo;m on holiday\u0026rdquo;, but I didn\u0026rsquo;t.\nKN: Another experience, right?\nUR: Lesson learned, I would say.\nKN:. So you have visited Poland a few times. SQLDay, SQLSaturday, etc. Which city is your favourite?\nUR: My favourite is Opole. OK, I know there are no conferences there, but my wife comes from there, the family is living there.\nKN: Yeah, because we should mention that your wife has a Polish name and has Polish roots.\nUR: She\u0026rsquo;s born Kućka. Her last name, Kućka. But the most beautiful city I was Wrocław. Wrocław is beautiful for two reasons: the city is very beautiful and the audience at the SQLDay, I love it. I absolutely love it. The venue is great and the hospitality is great, the people, the user groups, it\u0026rsquo;s absolutely brilliant. And everything counts a little bit. Krakow for example, we had a super, wonderful, beautiful speaker dinner there. Krakow with its Old Town, with the Wawel, so beautiful! It\u0026rsquo;s quite difficult to rate something, some city, which is better or not, but I absolutely prefer to go to Wrocław. Easy to get to, the aeroplane, and easy to fly back to Frankfurt. Warszawa - a metropole. It\u0026rsquo;s lovely but yeah, it\u0026rsquo;s a metropole.\nKN: How did it happen that you have a Polish wife?\nUR: Well, my wife, I know her from studying, from my studies. She\u0026rsquo;s living in Germany, oh my goodness, for over 40 years. And yeah, I was coming to Frankfurt because of my former girlfriend. We saw that it didn\u0026rsquo;t work with us, and at the university, I met my wife. And that was in 1988. In 1989 we became a couple. In 1991 I asked her to marry me and in 1992 we married. And since 1992 I\u0026rsquo;m one of the happiest husbands in the world. This is dedicated to my wife. I\u0026rsquo;m really happy with her. It\u0026rsquo;s beautiful. She tries as often as possible to come with me.\nKN: So you travel for the conferences together. And I\u0026rsquo;m not sure, most of the times or\u0026hellip;?\nUR: Unfortunately not. It depends on the location. So let\u0026rsquo;s say Scandinavia, Gothenburg, Stockholm, Oslo, she loves it, she absolutely loves it. But I don\u0026rsquo;t want to blame any city, but there are a few destinations where she said \u0026ldquo;Nah, I\u0026rsquo;m not so interested in\u0026rdquo; or she said, \u0026ldquo;I\u0026rsquo;ve been there so often\u0026rdquo;. For example, in Kyiv, she was twice in Kyiv with me. It\u0026rsquo;s a beautiful city, but she said: \u0026ldquo;OK, been there twice now, let\u0026rsquo;s have a one year break\u0026rdquo;. Same with Poland. I\u0026rsquo;m wondering about it. So I said \u0026ldquo;OK, come on, it\u0026rsquo;s SQLDay in Wrocław, will you join me?\u0026rdquo;, she said, \u0026ldquo;no, because one week before, two weeks before I will be there with mummy\u0026rdquo;. My mother-in-law, she\u0026rsquo;s 80 years old and sometimes they travel to Opole. And she said \u0026ldquo;no, I was over there a few weeks ago\u0026rdquo; or she had to work, she has a job. So she could not go every single time. But if there are some fine destinations like Madrid, South Africa, she tries to join me. What she doesn\u0026rsquo;t like to do is Asia. I\u0026rsquo;m addicted to Asia but she\u0026rsquo;s not. She was once with me in Singapore but that was only a stopover to the Philippines. She thinks Singapore is beautiful but it\u0026rsquo;s a big city. You cannot do very comfortable sightseeing, you don\u0026rsquo;t see old areas like you can see here in Madrid. That\u0026rsquo;s what she loves. But I\u0026rsquo;m really happy whenever she joins me. Really happy, I love it. It\u0026rsquo;s better to be together because we don\u0026rsquo;t see on the week. We don\u0026rsquo;t see each other in the week. So when I\u0026rsquo;m in Hamburg for example for a week, then I come back on Friday, and then on Friday it could be that I say \u0026ldquo;come on\u0026rdquo; and in the night we fly to Warszawa. So when we go to JOIN! conference in Warszawa in November. I only have two hours when I come back, two hours for switching my luggage, so I think it will be that my wife will come to the airport, we change the luggage, and I go back into the airport and she drives home. But she tries to join me as often as possible. But you need to keep in mind, another point I\u0026rsquo;ve forgotten is we have a family at home. My mother-in-law out there, her mother, her aunt is there, she has to take care of the house, of the garden, and that\u0026rsquo;s the reason why she said: \u0026ldquo;Uwe, go alone\u0026rdquo;. The good thing is she understands that I do it. She really understands. It is important and I\u0026rsquo;m so happy that she will join me when I go to New Zealand. OK, for her it\u0026rsquo;s the first time in New Zealand, but what she will do is she will go with me to the MVP Summit. But she knows that I completely off for four days. And I will come back only in the evening only. And in the evening, you know it best, there are some other parties going on. Meetings, parties in the Hyatt hotel, all this kind of stuff. But she said, \u0026ldquo;OK, I will join you\u0026rdquo;. I\u0026rsquo;m so thankful to her that she will do this. It\u0026rsquo;s impressive.\nKN: Yeah, I must admit. It\u0026rsquo;s not an easy thing to understand this kind of work.\nUR: But as long as she supports me, I do it.\nKN: And probably it\u0026rsquo;s easier with the fact that your children are adults right now, not living with you.\nUR: Yeah, the small one is 21. Caterina, she\u0026rsquo;s the oldest, she\u0026rsquo;s 25. They are living their lives. That\u0026rsquo;s cool. This means we have more time for us.\nKN: You\u0026rsquo;re not worried about them? What are they doing now?\nUR: No, I completely trust in what they are doing. I trust in everything that they are doing and for me, it is important that they know whenever they need help, they can come to me. And they know it. As I said, my family is my pole, it\u0026rsquo;s my south pole. They belong to me and that gives me the power.\nKN: That\u0026rsquo;s fantastic, I think it\u0026rsquo;s your biggest achievement.\nUR: It is, yes, it is. And I have a lovely wife. She was taking care of the family, of the kids, of our relationships throughout the years, because it\u0026rsquo;s really hard when I\u0026rsquo;m travelling all around the world. It\u0026rsquo;s really hard. But she was never complaining about it. Thank you, Bia! Dziękuję, Bia, dziękuję!\nKN: Yes. At the end of our conversation, could you tell us where people can find you on the Internet?\nUR: If you want to have a commercial thing, well if you go Google Uwe Ricken SQL Server, you will find me in Google. That\u0026rsquo;s the easiest way, but I\u0026rsquo;m on LinkedIn, but I\u0026rsquo;m not so active on social media. So Facebook is more private, for family. Twitter - to be honest, twittering is like chatting, you know? I don\u0026rsquo;t like it, not so much. Sometimes you will find a tweet from me when I\u0026rsquo;m complaining about something, but this covers only my current situation. When I have the feeling I have to tell the world what problem I currently have.\nKN: I haven\u0026rsquo;t seen you complaining about Thomas Cook on Twitter.\nUR: Yeah, but I was complaining about American Express because they didn\u0026rsquo;t give me my money back! You know what really sucks is that three days before Thomas Cook went bankrupt, they charged me 7500 euros! The rest of the value of the travel. God\u0026rsquo;s sake\u0026hellip; Good, it\u0026rsquo;s only for recording a voice but not seeing my face now. It makes me so angry! And what really makes me angry is you\u0026rsquo;re completely helpless. No one can help you. And you cannot do anything.\nKN: And you\u0026rsquo;re a business customer, you\u0026rsquo;ve been with them for a very long time.\nUR: Yeah, I\u0026rsquo;m a business customer. I\u0026rsquo;ve been a member of Amex for 30 years. But they don\u0026rsquo;t give a shit about it. Whatever. Don\u0026rsquo;t want to complain about it, we have other beautiful things. Facebook, LinkedIn, Twitter, that\u0026rsquo;s it. I hate Instagram, I hate this kind of stuff with followers, \u0026ldquo;oh I took a picture of myself\u0026rdquo;. It\u0026rsquo;s for young people. And I\u0026rsquo;m blogging but it\u0026rsquo;s in German, so mostly if you want to see me, go to SQLSaturdays, take the chance, visit other countries for SQLSaturday. I think it\u0026rsquo;s worth it that you open your mind for other cultures, for other languages, for other people. That is a gift for us speakers, for you and the same as me. You travel around and you have the opportunity to see different cultures. This is a gift that I really absolutely appreciate in my life.\nKN: I completely agree with you.\nUR: That\u0026rsquo;s what I love and that\u0026rsquo;s what I\u0026rsquo;m doing and that\u0026rsquo;s why I\u0026rsquo;m sitting here with you and say\u0026hellip; how do you say, \u0026ldquo;dzień dobry to all the Polish people\u0026rdquo;. I love it to sit here with you, to speak about simple things, not only technology, that\u0026rsquo;s great.\nKN: OK, well, thank you very much for joining me today.\nUR: Dziękuję bardzo i do widzenia!\nKN: Do widzenia, goodbye! Thanks!\nUseful links Uwe\u0026rsquo;s profiles: Twitter | LinkedIn\nUwe\u0026rsquo;s blog: Blog\nRelated events: SQL Saturday Madrid | Data Platform Geek Summit | Difinity | SQLDay\n","date":"2020-01-31T10:06:26Z","image":"/uploads/2020/01/ASF028-SQLSaturday-with-Uew-Ricken-1200.jpg","permalink":"/2020/01/asf-028-uwe-ricken-interview/","title":"ASF 028: Uwe Ricken interview"},{"content":"Good morning girls and boys, ladies and gentlemen, in the new year 2020! I recently thought that we had started a new decade. But that\u0026rsquo;s not true. The new decade begins next year, so we have to wait a moment.\nThis is not just another list of new year\u0026rsquo;s resolutions\u0026hellip; because I\u0026rsquo;m not a big fan of that kind of thing. Don\u0026rsquo;t get me wrong. That doesn\u0026rsquo;t mean that I\u0026rsquo;m not planning and having goals. I definitely have got them. I\u0026rsquo;m definitely a \u0026ldquo;planner\u0026rdquo; guy - love doing the plans, writing them down and checking what has happened after a certain period of time. What I meant is the period between Christmas and New Year Eve not particularly well moment to me to think about next year. That time is for me to relax and recharge the batteries.\nPast A few significant things happened for me in the previous year. One of those was a job change. I don\u0026rsquo;t want to repeat myself about the reason and circumstances because I describe it here. What I can say now, from time perspective (over 3 months now), it was a pretty good decision and I\u0026rsquo;m happy where I am. Obviously, I can not compare my current situation to that of what would happen otherwise, but so far so good.\nI managed to encourage my friend to join me on the blog. Michal Sadowski has been started writing posts in December 2018. We also have recorded a few episodes (5 so far) of the podcast together. Michal is experienced MCT as well as DBA, going towards a Data Engineer direction either. Recently he was writing about Docker and containers in this series. We have more plans for this year, hence I hope that his activity would even grow here. I still cannot understand why he is not an MVP yet, but I deeply believe that this will change in 2020.\nThe most-read posts (on this blog) in 2019 Like a year ago, also in this year, I would like to share with you the top 10* most popular articles visited during 2019. Here they are:\nHow to create DACPAC file? SCD type 1 \u0026amp; type 2 in MERGE statement Installation SQL Server client tools Unattended installation of SQL Server New DB project with SSDT and import existing database Deployment of Azure Data Factory with Azure DevOps Setting up Code Repository for Azure Data Factory v2 Azure Data Factory v2 and its available components in Data Flows Slowly Changing Dimensions (SCD) Mapping Data Flow in Azure Data Factory (v2) * Ranking based on Unique Page Views from Google Analytics.\nIf you follow me (@NowinskiK) or @SQLPlayer on Twitter - you might have remembered when I was pleased to spread out the word about that achievement, which I\u0026rsquo;m pretty proud of. Namely, we reached the traffic of 10k visitors a month.\nBoom! The number of blog visitors has broken the barrier of 10k per month. We wanna thank you to all visitors and as well as authors: @NowinskiK \u0026amp; @SadowskiMichal. Keep going! #mvpbuzz #Azure @SQLServer pic.twitter.com/yM6dosGgqh\n— SQLPlayer (@SQLPlayer) November 2, 2019 Podcast in numbers Hence, let\u0026rsquo;s reveal some numbers and statistics about the podcast (total):\n1,425 plays 3,897 downloads 27 listenable episodes published to date Over 19 hours of recorded conversations 42'43\u0026quot; is an average length of all episodes 13 different places where the chats have taken place 40,676 visits on interview\u0026rsquo;s pages The chart below comes from **Spreaker**where my podcast is hosted. The figures represent the whole year 2019.\nTwitter This is mainly for the record:\n@NowinskiK: 763 Following, 1142 Followers, 6715 Tweets.\n@SQLPlayer: 327 Following, 348 Followers, 1918 Tweets.\nYouTube At the end of the year 2019, I recorded the first technical video and published it on YouTube channel. Why? Two reasons. One: some things or steps are easier to explain by showing, hence an article is not an ideal way. Two: I hope I will take less time to prepare one video in contrary to write down a blog post. I hope because it isn\u0026rsquo;t a case yet. I need more practice, obviously.\nSo, check out my first video and stay tuned to watch more in 2020:\nSQLPlayer channel on YouTube.\nMicrosoft Certificates I passed the exam on December 31st. It was fun. I had only a few days for preparations and my attitude was like: Let\u0026rsquo;s take the exam and see what kind of questions it will bring. Probably I\u0026rsquo;d fail, then prepare better and attempt again. But I passed at the first attempt. Even more, I was feeling that during the exam. You know, such feeling like \u0026ldquo;not so bad, not so bad. There is a chance\u0026rdquo;.\nWhat could be better than getting up at 6am on December 31st and taking the online exam for the first time from home? It may only be better to pass this exam :) This is how the year ends! ?? pic.twitter.com/njZRx9uFaJ\n— Kamil Nowinski (@NowinskiK) December 31, 2019 Plan for 2020 I must disappoint you here. I\u0026rsquo;m not gonna make any new year\u0026rsquo;s resolution. After a passing year of experience and thoughts - I built a shortlist of ideas for the incoming year. As usual, this is an initial list and will be evolving during the year.\nLess conference\nI will try to take part in fewer conferences comparing to 2019. Or at least not more. All in order to save time and focus on\u0026hellip;\nMore blogging \u0026amp; making videos Yes, this is the next year when I\u0026rsquo;m planning to focus on the blog. I\u0026rsquo;m really pleased to develop that point and type of knowledge. Also, I glad to see that the blog is growing (by the number of visitors). I will be doing that as long as it makes me happy.\nMore exams \u0026amp; certifications\nOn the really last day of 2019, I passed the exam AZ-203. It was over 3 years since my last Microsoft exam is taken. I had no clear path for exams during 2019, but I decided to change this in 2020. This is because I have that gap and also, working with Azure services a lot - it will help me to verify \u0026amp; extend my knowledge and confirm my qualifications.\nOne projectThis year, I\u0026rsquo;m planning to build one project. Completely new to me. I don\u0026rsquo;t want to reveal the details, but I will when finish. It\u0026rsquo;s a secret yet and only insiders know.\nHappy New Year! * This is me. And what about you? What\u0026rsquo;re your plans, dreams, goals? Share your thoughts with me and leave a comment. Whether you do it or not - I wish you\nHappy New Year! Fight for every millimeter of happiness and remember that only a happy person can make others happy.\n* Originally, I had a plan on the 2nd of January to release this post. But it has finally happened, even though a bit late.\n","date":"2020-01-10T21:12:35Z","image":"/uploads/2020/01/joshua-earle-ICE_bo2Vws-unsplash-1200px.jpg","permalink":"/2020/01/the-recap-of-the-passing-year-2019/","title":"The recap of the passing year 2019"},{"content":"Hello boys \u0026amp; girls! This is the last post in \u0026ldquo;Last Week Reading\u0026rdquo; series in this decade. Can you believe that?\nHave you had a good Christmas time? Have you had time for reading and learning? Twice \u0026ldquo;Yes\u0026rdquo; here.\nOoouuu\u0026hellip; Happy New Year 2020! I wish you all your plans come true!!!\nPress Series: Beginner\u0026rsquo;s Guide to Azure Data Factory\nCathrine wrote 25 posts in December. She is amazing.\nBest Practices for Implementing Azure Data Factory\nHow Paul does it.\nFifteenth Blog Birthday\nChris Webb summaries last year.\nSQLDay 2019 - sessions (video)\nAll recorded session will be available over there soon.\nSnowflake for SQL Server Users\nDr Greg Low\u0026rsquo;s eBook is available now. First part first :)\nPower BI Helper December 2019 Feature Summary\nThe latest version of the tool is available. Thanks Reza!\nRedgate: The 2019 State of SQL Server monitoring\nLearn how 820 SQL Server professionals manage their estates.\nTool: CloudSkew\nMithun is writing an app that\u0026rsquo;ll help you create azure architecture diagrams for free.\nThe following articles/docs might be useful for those who prepare for exam AZ-203:\nCreate or update custom roles for Azure resources using Azure PowerShell\nAPI Management caching policies\nASP.NET Session State Provider for Azure Cache for Redis\nTrack custom operations with Application Insights .NET SDK\nAzure Service Bus Topic And Subscription (Pub-Sub)\nNotification Hubs: Registration management\nGitHub: Azure Search Jobs Website Sample using ASP.NET MVC\nHow to implement faceted navigation in Azure Cognitive Search\nHow to use Azure Cognitive Search from a .NET Application\nAzure Search, The basics\nA word from (former) Microsoft’s CEO What’s next for our foundation? I’m particularly excited about what the next year could mean for one of the best buys in global health: vaccines. https://t.co/MbjNWPEl5l\n— Bill Gates (@BillGates) December 19, 2019 Video: Azure Virtual Machine Introduction https://www.youtube.com/watch?v=iUaTq06m26g\nSmile Corner ","date":"2019-12-30T20:37:09Z","image":"/uploads/2019/12/new-years-eve-2019-QAhSBj237_w-unsplash_1200px.jpg","permalink":"/2019/12/last-week-reading-2019-12-29/","title":"Last Week Reading (2019-12-29)"},{"content":"Hello. It\u0026rsquo;s been a while since the last post of this series appeared. I will do my best to keep this going. So, without unnecessary introductions - let\u0026rsquo;s move on.\nPress Power BI Desktop December 2019 Feature Summary\nKPI visual formatting settings is one of the most interesting features this month.\nHow Microsoft re-envisioned the data warehouse with Azure Synapse Analytics\nA new service that merges the capabilities of Azure SQL Data Warehouse with new enhancements such as on-demand query as a service.\nhttps://www.slideshare.net/jamserra/azure-synapse-analytics-overview\nMicrosoft Ignite 2019\nAll the videos from the conference.\nPawel Potasinski about Snowflake.\nMaking Java Spring Boot (with containerization) work with Azure DevOps\nEnd-to-end DevOps works with AKS, ACR, Docker.\nSQL Server 2019 Intelligent Performance - Worker Migration\nProcess AKA “worker stealing”.\nAzure Advent Calendar\nYouTube channel with 20-30 minutes of videos about various Azure topics.\nPostman - using variables\nNeeds to test REST API of a web app or Cosmos DB? This tool is awesome.\nVideo: Tips on partition optimizations in ADF Mapping Data Flows.\nDeep dive: Mission-critical multi-tenant apps with Cosmos DB multi-master\nVideo: Multi-tenant concept with Cosmos DB.\nThe Azure Certifications For 2019\nCertification path for everyone who would like to pass the exam(s) - maybe even this year?\n25 days of serverless\nEvery day new challenge.\nA word from Microsoft’s CEO One of the great privileges of my job is meeting people who are applying technology to have an impact. As I reflect on 2019, here’s a look at 10 people who inspired me by exemplifying how technology can be used to create new opportunity for everyone. https://t.co/kYFwjPlXEl\n— Satya Nadella (@satyanadella) December 18, 2019 Video: Using Azure DevOps for Azure Data Factory https://www.youtube.com/watch?v=uflQ3nmXHhc\nSmile Corner ","date":"2019-12-22T20:09:26Z","image":"/uploads/2019/11/news-tyler-nix-tPz-uoz5RBU-unsplash-1200.jpg","permalink":"/2019/12/last-week-reading-2019-12-22/","title":"Last Week Reading (2019-12-22)"},{"content":"Hello, data folks!\nI have published several articles about databases and their maintenance in SSDT project on the blog. One of the posts explained step by step how to import a database from a production environment and thus create the SSDT project.\nIn this video, I will explain to you how to set up build and release pipelines in Azure DevOps, so we could deploy and publish SSDT database project to our target SQL Server instance incrementally.\nThis is a very simple example only - how to do that with a minimum number of steps.\nBut in the next videos, I will show you how to extend it by adding more and more features.\nStay tuned!\n","date":"2019-12-12T01:09:20Z","image":"/uploads/2019/12/DevOps01.jpg","permalink":"/2019/12/deployment-of-microsoft-sql-database-with-azure-devops/","title":"Deployment of Microsoft SQL database with Azure DevOps"},{"content":"This is the second post on series related to the installation and configuration of Docker containers on the Microsoft Windows platform.\nThis series is for professionals who start their journey with the containers and use them with Microsoft SQL Server.\nThe goals that we would like to achieve today are as follows:\nDownload SQL Server images to be used with Docker Run SQL Server on Docker Configure more than one SQL Server instance on Docker Access SQL Server instance using tools like SQL Server Management Studio and Azure Data Studio Restore database to SQL Server on Docker Perform database upgrade between different Cumulative Updates levels and upgrade from SQL Server 2017 to SQL Server 2019 Preparations We are starting with the following assumptions:\nAll steps described in this first blog post has been configured. The computer, where Docker is installed, has access to broadband Internet (needed for downloading Docker images) Download SQL Server images to be used with Docker As a first step, we need to check what SQL Server images are available on the Microsoft website. To check this I will run following PowerShell command:\ninvoke-webrequest https://mcr.microsoft.com/v2/mssql/server/tags/list After running above command, you will get the following output:\nAs a second step, I will assign the output of the command to a variable and check Content property:\n$repository = invoke-webrequest https://mcr.microsoft.com/v2/mssql/server/tags/list $repository.Content Following output should be shown:\nIf you want to download a specific version of SQL Server binaries (2017 with Cumulative Update 11 in our case), you need to run following command:\ndocker pull mcr.microsoft.com/mssql/server:2017-CU11 After some time (depending on your network connection), you should get the following output:\nAs we have images downloaded already, we can go to the next step - running first SQL Server on Docker\nRun SQL Server on Docker To run SQL Server on Docker, you need to provide a list of parameters that allows SQL Server to use proper settings and communicate with the external world outside Docker.\ndocker run -e \u0026#39;ACCEPT_EULA=Y\u0026#39; -e \u0026#39;MSSQL_SA_PASSWORD=SecretP@ssw0rd\u0026#39; ` --hostname sql2017cu10 ` -p 1401:1433 ` -v sqlvolume:/var/opt/mssql ` --name sql2017cu10 -d mcr.microsoft.com/mssql/server:2017-CU10-ubuntu Below there is an explanation of used parameters:\n-e (or -env) - sets environment variables, in our case we accept End User License Agreement and sets a password for \u0026ldquo;sa\u0026rdquo; --hostname - sets container hostname, we can omit this, then docker will generate its own name -p (or --publish) - publishes a container’s port to the host. In our case default SQL port (1433) is published on port 1401 -v (or --volume) - binds mount a volume. It is the location of persistent storage for SQL Server databases --name - assigns a name to the container -d (or--detach ) - runs the container in background and print container ID After a few seconds, a new container with SQL Server should be up and running.\nTo check the status of containers you can run the following command:\ndocker ps -a You should see the following output:\nLet\u0026rsquo;s check if T-SQL commands can be executed against Docker container:\nsqlcmd \u0026#39;-Usa\u0026#39; \u0026#39;-Slocalhost,1401\u0026#39; \u0026#39;-Q\u0026#34;SELECT @@VERSION\u0026#34;\u0026#39; \u0026#39;-PSecretP@ssw0rd\u0026#39; You should see the following output:\nAs you can see we have created a container with a version of SQL Server that we have specified is up and running.\nRestore database to SQL Server on Docker As a next step is to copy database backup to Docker container and restore this backup.\nFor coping database backup, I am running docker cp command:\ndocker cp C:\\SQLBackup\\WideWorldImporters-Full.bak sql2017cu10:/var/opt/mssql After a few seconds, a backup should be successfully copied to the container:\nHaving a backup inside the container we can run T-SQL command to restore a backup to the volume specified during the start of container:\ndocker exec sql2017cu10 /opt/mssql-tools/bin/sqlcmd ` -S localhost ` -U sa ` -P \u0026#39;SecretP@ssw0rd\u0026#39; ` -Q\u0026#34;` RESTORE DATABASE WideWorldImporters ` FROM DISK = \u0026#39;/var/opt/mssql/WideWorldImporters-Full.bak\u0026#39; ` WITH MOVE \u0026#39;WWI_Primary\u0026#39; TO \u0026#39;/var/opt/mssql/data/WideWorldImporters.mdf\u0026#39;, ` MOVE \u0026#39;WWI_UserData\u0026#39; TO \u0026#39;/var/opt/mssql/data/WideWorldImporters_userdata.ndf\u0026#39;, ` MOVE \u0026#39;WWI_Log\u0026#39; TO \u0026#39;/var/opt/mssql/data/WideWorldImporters.ldf\u0026#39;, ` MOVE \u0026#39;WWI_InMemory_Data_1\u0026#39; TO \u0026#39;/var/opt/mssql/data/WideWorldImporters_InMemory_Data_1\u0026#39;\u0026#34; After a few seconds, the familiar output should be visible saying that database backup has been successfully restored.\nAt this point, we have a container running SQL Server 2017 CU10 with database WideWorldImporters restored.\nConfigure more than one SQL Server instance on Docker Next step is to run the second container of SQL Server with the latest version of SQL Server (CU17 at the time of creating this blog post)\ndocker run -e \u0026#39;ACCEPT_EULA=Y\u0026#39; -e \u0026#39;MSSQL_SA_PASSWORD=SecretP@ssw0rd\u0026#39; ` --hostname sql2017latest ` -p 1402:1433 ` -v sqlvolume2:/var/opt/mssql ` --name sql2017latest ` -d mcr.microsoft.com/mssql/server:2017-latest There are few changes comparing to the first container:\nhostname is different to match SQL Server binaries port is set to 1402, so we can connect to both instances if needed volume is configured to use different that first container name is different to match SQL Server binaries The second container has been created, let\u0026rsquo;s check what is running on Docker (we expect to have to containers running):\ndocker ps -a You should see the following output:\nOur environment is up running, the next step is to check if both containers behave the same as \u0026ldquo;old-school\u0026rdquo; SQL Server on Windows. To check that we are going to use SQL Server Management Studio and Azure Data Studio\nAccess SQL Server instance using tools like SQL Server Management Studio and Azure Data Studio The first tool that I am going to use is SQL Server Management Studio. After starting SSMS, I am trying to connect to the first container configured to listen on port 1401\nAfter clicking on Connect, you are connected to the container with SQL Server\nProbably you have noticed that icon of the instance is different (with the penguin as Docker is Linux based) and version of the binaries (after checking on this page you will find it is CU10).\nLet\u0026rsquo;s try to run some basic queries on the instance and database levels:\nUsing T-SQL to check the version of SQL Server:\nLet\u0026rsquo;s check WideWorldImporters database:\nTill now all queries behave the same way as on the SQL Server installed within Microsoft Windows environment.\nThe only difference that you can find (except different icon) is visible within properties of the instance:\nLet\u0026rsquo;s check the Azure Data Studio. As a first step, let\u0026rsquo;s connect to the container:\nNow create a new notebook:\nCreate the first Text cell with description and add code with T-SQL statements\nYou can execute T-SQL statements by clicking on Run cells or by selecting a single cell and pressing F5\nAs of now, we have created and started a container where SQL Server is running with the user database.\nThe next step is to perform database update to a higher version of SQL Server binaries\nPerform database update between different Cumulative Updates levels and upgrade from SQL Server 2017 to SQL Server 2019 Till now we have used two containers with two versions of SQL Server binaries (CU10 and CU17). The aim of this paragraph is to show a situation when you would like to perform compatibility tests of your database with a higher version of SQL Server binaries - testing before actual update to the latest version.\nHigh-level steps look as follows:\nStop container with the current version of SQL Server binaries Stop container with the latest version of SQL Server binaries Attach volume with a database to the container with the latest version of SQL Server binaries Perform tests on the database that is running on the latest version of SQL Server binaries Let\u0026rsquo;s start with stopping the container with SQL Server 2017 CU10. To achieve this run following command:\ndocker stop sql2017cu10 Let\u0026rsquo;s create a new container for our test\ndocker run -e \u0026#39;ACCEPT_EULA=Y\u0026#39; -e \u0026#39;MSSQL_SA_PASSWORD=SecretP@ssw0rd\u0026#39; ` --hostname sql2017test ` -p 1401:1433 ` -v sqlvolume:/var/opt/mssql ` --name sql2017test ` -d mcr.microsoft.com/mssql/server:2017-latest Process of the startup of a new container takes maximum few seconds\nHowever, when you try to connect to an instance, you will get the following message:\nThis message clearly says why you can\u0026rsquo;t connect to the instance - just after the startup of SQL Server, an update of the database is performed to the same version as the version of the master database (thus version of the database instance). After some time (in my case it was about 40 seconds), you can normally connect to the instance:\nVersion 14.0.32381.1 clearly indicates that SQL Server is running on Cumulative Update 17 binaries.\nAfter examination of SQL Server Log, you will discover that upgrade script has been executed to an instance to update it to mentioned CU level\nAs you can see, performing an update from one CU level to the other is quite straight-forward.\nNow, let\u0026rsquo;s check if it is possible to upgrade our database from SQL Server 2017 to SQL Server 2019. To achieve this, high-level plans look similar to the previous one, with the exception that a new container will be using SQL Server 2019.\nLet\u0026rsquo;s start with stoping currently running container:\ndocker stop sql2017test There is no output, as shown below\nLet\u0026rsquo;s run a container with SQL Server 2019 by executing the following command\ndocker run -e \u0026#39;ACCEPT_EULA=Y\u0026#39; -e \u0026#39;MSSQL_SA_PASSWORD=SecretP@ssw0rd\u0026#39; ` --hostname sql2019 ` -p 1402:1433 ` -v sqlvolume:/var/opt/mssql ` --name sql2019 ` -d mcr.microsoft.com/mssql/server:2019-latest So far, so good, the new container has started:\nHowever, when you check the status of the container, you will notice that container with SQL Server 2019 is stopped:\nWould be nice to have the option to check what is the reason for this situation. There is a handy command to check logs of the container:\ndocker logs sql2019 After execution of this command, it starts to be more clear why SQL Server has been stopped just after start:\nIn the output of the command, there is a link to Microsoft\u0026rsquo;s website with a possible solution to the issue.\nI have found that solution below fixed my issue:\n1. Run the previous version of a container (SQL Server 2017 CU 17 in my case)\n2. Change ownership of the directory where databases are stored to allow users in the root group to access them in the built image\n3. Copy access permissions on the directory where databases are stored from user to group\ndocker start sql2017test docker exec -it sql2017test \u0026#34;bash\u0026#34; chgrp -R 0 /var/opt/mssql chmod -R g=u /var/opt/mssql exit Commands should be executed with any issues\nLet\u0026rsquo;s try to upgrade to SQL Server 2019\ndocker stop sql2017test docker start sql2019 docker ps -a The expected output is shown below\nEverything looks good, let\u0026rsquo;s check our instance and database using notebook created earlier\nQuery executed on instance level show correct build version (expected 2019), let\u0026rsquo;s check our database:\nHere also everything works as expected.\nAs a last try let\u0026rsquo;s check if it is possible to downgrade our database from SQL Server 2019 to SQL Server 2017. To achieve this we need to execute the following steps:\nStop container with SQL Server 2019 Run container with SQL Server 2017 Try to connect to instance and query instance and database Following commands needs to be executed\ndocker stop sql2019 docker start sql2017test docker ps -a When you have look at the output, you will see that the container was up and running for a few seconds and then stopped.\nAs learned earlier, let\u0026rsquo;s check what is within logs of the container by running this command\ndocker logs sql2017test At the beginning of the log, everything looks normally\nBut at the very end, there is an explanation of the container shutdown\nAs stated in the message, you can only upgrade the database, a downgrade is not possible.\nClean-up After all activities that we have performed on the Docker, sometimes it is required to clean-up resources (both running containers burning CPU and images taking disc space). As a first step let\u0026rsquo;s list all containers\ndocker ps -a In my case all containers are stopped:\nYou can remove containers by providing name or container-id\ndocker rm sql2017downgrade docker rm c182dcc1fa36 No output means that commands have been executed successfully\nIn my case, there were three more containers to be removed\ndocker rm sql2017cu10 docker rm sql2017latest docker rm sql2017test After executing these commands, I don\u0026rsquo;t have any container on my Docker.\nNext step is to remove volumes created during container startup\ndocker volume rm sqlvolume docker volume rm sqlvolume2 Execution of the command should take less than a second\nThe last step is to remove images downloaded from the repository. Let\u0026rsquo;s check if there are any images downloaded already\ndocker images On my machine, there are the following images\nTo remove image just run following command\ndocker rmi ba266fae5320 Output looks following\nYou can repeat the same command for the other images.\nSummary As you can see using containers available within Docker gives many possibilities of the rapid development of code on a different version of SQL Server binaries without a need to the tedious process of setting up the virtual machine and configuration of SQL Server instance. These possibilities allow developers to use their own workstation and develop required changes at a much faster pace compared to the previous approach with requesting a refresh of the lower region using the latest Production backup (that can take days instead of minutes on Docker). Docker gives also much lower overhead compared to a virtual machine thus is a perfect solution that should be considered by every developer.\nLinks SQL Server 2017 build versions\nBuild and run non-root SQL Server 2017 containers\n","date":"2019-12-10T08:00:45Z","image":"/uploads/2019/12/white-water-boat-753331.jpg","permalink":"/2019/12/using-sql-server-on-docker/","title":"Using  SQL Server on Docker"},{"content":"It is the first post on series related to installation and configuration of Docker containers on the Microsoft Windows platform.\nThis series is for professionals who start their journey with the containers and use them with Microsoft SQL Server.\nToday\u0026rsquo;s aims to complete the following steps:\nDownload and install Docker Download and configure Git Clone Docker repository locally Check if Docker is running correctly after installation Preparations We are starting with the following assumptions:\nDocker will be installed on Windows 10 64-bit: Pro, Enterprise, or Education (Build 15063 or later). The following hardware prerequisites are required to successfully run Client Hyper-V on Windows 10: 64-bit processor with Second Level Address Translation (SLAT) 4GB system RAM BIOS-level hardware virtualization support must be enabled in the BIOS settings. For more information, see Virtualization. Computer with Internet access Hyper-V and Containers Windows features must be enabled To check if Hyper-V is enabled go to Start -\u0026gt; Turn Windows features on or off\nCheck if Hyper-V Hypervisor is enabled\nDocker Installation Open your favourite web browser and go to www.docker.com web page and click on Sign In.\nCreate a new account and login to it. Once you are logged in, you should be able to see the welcome page. Click on Download Docker Desktop for Windows\nA new web page will be opened with link to Docker binaries - Download Docker Desktop for Windows\nOnce the file is downloaded, double click on it so installation can start. Use default options as on the screen below\nClick Ok and wait a few minutes for the installation to be completed.\nOnce Docker is installed you need to log out and log in again. For best experience restart your computer.\nOnce you are logged in again, go to Start and run PowerShell as Administrator\nRun following Powershell command to clone GitHub repository locally:\ngit clone https://github.com/docker/doodle.git If you didn\u0026rsquo;t have a chance to play with git before, you will probably get the following error:\nBelow there are steps how to download and configure Git\nGit installation To solve this issue go to Git webpage and download the latest version of binaries\nBinaries should start downloading within a few seconds, you can save them or run after download:\nAfter a few seconds, (depending on your network bandwidth) installation of Git should start. Read the License and click Next.\nSelect a location where Git binaries should be installed, in my case, I took the default one. Click Next to go to the next configuration screen\nOn \u0026ldquo;Select Components\u0026rdquo; accept the default components and click Next\nOn \u0026ldquo;Select Start Menu Folder\u0026rdquo; click Next\nOn the \u0026ldquo;Choosing the default editor used by Git\u0026rdquo;, change default one to Notepad++. If you don\u0026rsquo;t have Notepad++ installed, you can download it from this webpage\nOn \u0026ldquo;Adjusting your PATH environment\u0026rdquo; leave the default option and click Next\nOn \u0026ldquo;Choosing HTTPS transport backend\u0026rdquo;, select Use the OpenSSL library. If you are working within a corporate environment with Certification Authority configured, select the other option\nOn \u0026ldquo;Configuring the line ending conversions\u0026rdquo; leave default option and click Next\nOn \u0026ldquo;Configuring the terminal emulator to use with Git Bash\u0026rdquo; use MinTTY to avoid some issues of default Windows\u0026rsquo; console window\nOn \u0026ldquo;Configuring extra options\u0026rdquo; leave the default selection and click Next\nOn \u0026ldquo;Configuring experimental options\u0026rdquo; leave the default selection and click Next\nWait a few minutes for the installation to be completed\nDeselect View Release Notes and click Finish\nCongratulations! You have successfully installed Git.\nRunning the first container Re-run PowerShell console and try to run the following commands:\ncd\\\rmkdir Git\rcd Git\rgit clone https://github.com/docker/doodle.git Run command to build sample solution\ncd doodle\\cheers2019 ;\rdocker build -t \u0026lt;your_id\u0026gt;/cheers2019 . There should be the following output\nFrom new PowerShell window, run following command\ndocker run -it --rm \u0026lt;your_id\u0026gt;/cheers201 Settings of Docker In the taskbar you should be able to see Docker icon and hovering over it should show the status of Docker. After right click on the Docker icon you can go to Settings for configuration of Docker VM:\nIn the Advanced settings tab, you can configure the CPU and Memory configuration of Docker\nNote: all of the above settings, are applied to VM visible in Hyper-V Manager\nAfter right click on DockerDesktopVM and selecting Settings, the following window will appear\nAfter all these steps, you have successfully installed and configured Docker and Git components.\nSummary In this blog post, I have shown step-by-step instructions on how to download, install and configure Docker and Git components.\nHope you will find these instructions helpful and allows you to start with Docker.\nNext blog post will describe how we can use freshly installed Docker with Microsoft SQL Server.\nLinks Docker binaries for Windows - https://download.docker.com/win/stable/Docker%20Desktop%20Installer.exe\nGit - https://git-scm.com/downloads\nDocker GitHub repository - https://github.com/docker/doodle.git\n","date":"2019-12-03T08:00:03Z","image":"/uploads/2019/11/birds-eye-view-photo-of-freight-containers-2226458.jpg","permalink":"/2019/12/installation-of-docker-desktop-for-windows/","title":"Installation of Docker Desktop for Windows"},{"content":" Introduction Mikael Wedham is the first Microsoft Certified Master on SQL Server 2008 in Sweden. He has worked as a developer and database administrator since 1993 and in \u0026lsquo;97, Mikael became a Microsoft Certified Trainer. He has gone through a pedagogical education with DSAT. Mikael was one of the first in Sweden giving lectures in Microsoft OLAP Services for SQL 7.0 and is also hired as Business Intelligence expert. Mikael is the founder of SQL Saturday in Sweden, where he also appears as an appreciated speaker.\nThis talk has taken place after SQL Saturday 2019 in Gothenburg, Sweden on 15 September 2019 (Sunday).\nInterviewer: Kamil Nowinski (T).\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript Kamil Nowinski: At the beginning, could you introduce yourself and where you live?\nMikael Wedham: My name is Mikael Wedham, I live in Växjö, Sweden. It’s a city with 70,000 people, in the centre south of Sweden.\nKN: Fantastic, so currently we are sitting in a coffee bar and Gothenburg, right?\nMW: Yes.\nKN: Could you tell me and us what happened yesterday?\nMW: Oh, we had the 5th SQLSaturday in Gothenburg. In Gothenburg, since that’s my hometown, I grew up here. And I’ve been arranging it for five years now.\nKN: So how many tracks were there yesterday and how much effort does it cost you to prepare something like SQLSaturday?\nMW: Yesterday we had six slots during the day or five separate tracks but we had them in three rooms so we had different subjects in different rooms. And I’ve been preparing basically from November, a couple of hours every week and some weeks maybe 40 hours.\nKN: Started in November?\nMW: Yes.\nKN: Oh, it’s almost like a year ago.\nMW: Yeah. So next year’s SQLSaturday we’ve already started a month ago to prepare for that one. So it takes a lot of planning and a lot of preparations to do that.\nKN: And also you speaking a lot during that kind of conferences, right?\nMW: Yeah, I try to go to almost one SQLSaturday every month.\nKN: Fantastic. Could you share your experiences in that area? How do you prepare for speeches and how does it work for you?\nMW: Oh, that’s kind of a secret, because I don’t prepare that much. I’m a SQL Master [Microsoft Certified Master], so I think I have pretty good knowledge on the things, and then when I do my presentations, I usually find the area I want to do a presentation about, and then I may be practising and put some slides together, do some demos, and then I go and do the presentation. Because, as everybody who presents knows, the presentation is never going as planned. So I don’t plan that much. I just do what works in the presentation and sometimes things work and sometimes they don’t.\nKN: OK, so you mean that sometimes you improvise?\nMW: Yes, that can happen. We discussed that yesterday that I never touch my presentation or demos the week before I do the presentation because I don’t want to break something. But sometimes things break themselves, so\u0026hellip;\nKN: Especially these days when everything changes almost every day in Azure.\nMW: Yeah, when you discover new APIs and your APIs are gone, that’s the time for improvisation.\nKN: Yeah, also it’s a little tougher to present something with Azure services because you need access to the Internet, right? And it’s not always the case.\nMW: That was a challenge yesterday because we’re at the public school and they have regulations about who gets access to the Internet. And we don’t get tickets or passwords for all the speakers. So I had to tether my phone and I had one of my colleagues to tether phones, to give speakers Internet access to do that stuff.\nKN: But nothing bad happened yesterday, right?\nMW: I heard no one reporting no internet connection. Everything was perfect, apart from that we had a power failure of the entire venue, just five minutes before we started.\nKN: OK, I didn’t notice that!\nMW: But that’s like a mishap. It was covered pretty well.\nKN: OK, you mentioned SQL Master. Could you explain that for people who don’t know what that is?\nMW: When you do a normal certification, you get a certification obviously and then you can do a lot of certifications. Then you will have higher titles, like Solution Developer or Systems Engineer, Database Developer, and that’s usually two, three, four certifications. But then, a couple of years ago, Microsoft also had intensive deep training with a certification that was totally crazy. And since I’m also totally crazy, I thought it was a good idea to do that certification, and I passed. So I was one of the lucky few that made that certification before it got cancelled.\nKN: So, to summarize, it’s MCM, right?\nMW: MCM, yes.\nKN: So how did you prepare for that? I’m guessing that it was very hard to pass that exam.\nMW: Yes. First thing, I prepared for almost a year all waking hours, I wrote my own course material, I prepared by reading blogs, talking to people, doing practice things. So I was extremely well prepared. And then, when I came to the certification, I realized that I got tasks that usually I do in one week, and they said “you have eight hours” and then, when I had done this certification, I was so empty and tired, I almost cried. And I failed. And then I prepared even more, but\u0026hellip;\nKN: So you failed the first time?\nMW: Yes, I failed the first time and then I prepared even more. But the most preparation was actually probably I knew that I was never going to have time to do everything. So I mentally prepared to fail and that made me more relaxed, as in “OK, I’m gonna do as much as I can, and the things I don’t have time to do, I can’t do them”. So the second time I did it, the same feeling, I almost cried, and I got the message a couple of weeks later that I passed.\nKN: Fantastic. But the first time it had to be very hard because you thought you were very well prepared, unfortunately, the result was\u0026hellip;\nMW: Nothing can prepare you for that experience. It was so tough, so broad, so wide.\nKN: So how do you prepare better for the future, if you were to give some advice to other people? I know that MCM doesn’t exist nowadays but if people are preparing for other exams.\nMW: You need to work with things more, and know more how things work, use books online, and experiment. See that the things work like they should find things that aren’t like they should.\nKN: So read, practice, etc.?\nMW: Yeah, know best practices, know better practices and work with all parts of the product.\nKN: OK, let’s talk about what you do for a living, on a daily basis.\nMW: Right now I’m around half time Managing Director of SolidQ in Sweden, and the other half is consulting for SolidQ Sweden. So I’m sharing my job, so I’m a full-time employee at SolidQ but doing consulting and managing director things in a mix.\nKN: So this is a part of your job?\nMW: Yes, my job is part-time consulting, part-time doing managing director things.\nKN: How does it work for you? Or maybe I should ask differently first: when did you start with SolidQ?\nMW: I started with SolidQ as a subcontractor and CTO in 2012, but I changed to being an employee this 1st of December last year.\nKN: OK, so something quite new. Why did you decide to go in that direction?\nMW: Because they gave me the famous offer you can’t refuse. I got an offer to be Managing Director and I told them: “but I don’t like managing budgets and invoicing and financial things because that’s boring” and the board said that we knew you were gonna say that, so I got the opportunity to have Torben as my help to do that stuff I don’t like. So I actually do the CTO things, managing work, and do the things I’m good at, and still get to do consulting. So it was a no-brainer. I got to work with SolidQ as I was before.\nKN: So they basically gave you a very good offer. So do you think that the situation that you are doing something more, something extra apart from your normal job or getting out of your comfort zone could help give you that kind of offer?\nMW: Yes, because before I had a lot of customers and I had only myself to work for the customers, so I started to give a job to my friends at SolidQ and make them work for me as a subcontractor. And it was so much work to keep track of “OK, whose customer is this, whose customer is that”. So we’re actually saving time by me keeping the customers but letting my friends and me work at the same place. So we’re more as one company.\nKN: That sounds good. What do you think about the MVP, MVP title and award?\nMW: Sometimes I’m thinking about MVP but I haven’t\u0026hellip; I have been nominated but I haven’t been approved. And I’m thinking that if I am going to be an MVP, I want to be an MVP because I do what I love, and I am never gonna change what I do to get the MVP title. So I’m not gonna start blogging 16 hours a week just because that’s what it takes. So if I become an MVP, that’s gonna be wonderful. I would love it probably, but if I don’t become an MVP, that’s because I don’t love the things that it takes to be an MVP.\nKN: That the general goal, do what you love doing and one day it might happen.\nMW: You never know.\nKN: Exactly. What about your work-life balance? I know that it’s quite hard to do some extra things like organizing events, like going and doing some extra work, like giving a speech during the conference. Because it’s our voluntary time. So how can you manage that work-life balance?\nMW: I have a very understanding wife and we decided already in 2004 that my way of working is that I’m away from home quite a lot but then when I’m at home, I’m fully at home. So I work from home during un-customer hours or un-customer weeks, because in Sweden we have this public holiday in the summer, it’s maybe for me usually six to eight weeks where there are not many customer requests. That’s when I can be at home, do administration stuff and do things that everybody else does during the year. So I have a big pile of papers and stuff that I must manage. But I am still at home. So we have a good compromise and a good balance there. And then usually I go diving with my kids a couple of times a year. And I try to sing in a choir as much as I can.\nKN: You have children, right?\nMW: Yes, three.\nKN: What ages?\nMW: 24, 18 and 16.\nKN: So now they are more mature and they can manage their lives more.\nMW: Yeah, so my oldest daughter lives in Gothenburg and she’s assisting in the SQLSaturday, so she is one of the volunteers.\nKN: Oh, so she was sitting with us yesterday during lunch, right?\nMW: Yes. And also my mother as one of the volunteers.\nKN: That was my understanding that it’s your mother. So fantastic, almost the whole family, let’s say half of your family engaged in organising the SQL event. So for sure, we can say “SQL family”, right?\nMW: Yes, yes it is.\nKN: That’s fantastic. So what do you think is the hardest these days in technology or in the area that you are working with?\nMW: It’s interesting, the hardest is that when I was 25 I was an expert, so I knew everything and I went to customers maybe some time and telling the customers “you can only do this in one way and it is this way that I tell you”, and the customer said “OK, thank you for your information, we’ll ask three other consultants”, to make sure. And I was sometimes a bit annoyed that “oh, why aren’t you doing this, because this is right, I know it”. I still have that issue that when I know I am right, I might be a bit hard to being talked into that I’m not right. But now when I’m unsure, I go to the customer and the customer asked me “OK, what do you think about Power BI?” and I tell the customer “I have no clue, I have installed Power BI once and I think you can use it like this” and I hear the customer goes and says “well, Michael said we could do that this way because he has been doing that” and I’m like “no, I’m unsure, I have no clue”, but now when I’m mature, at least I look mature, then the customer trusts me more. So the hard thing is to know when to push, “OK, this is a thing I know for a fact” and to choose those things. Because a lot of the things in our field is connected, so I realized I need to know Azure, I need to know Storage, I need to know SAN, I need to know Virtual Machines, but I can only be the best on so many things. So what do I choose?\nKN: Exactly, so that’s the reason why you are unsure of some solutions, right? You can feel some solution because of the knowledge, right?\nMW: Yes, I know how it should be, but I’m not sure anymore that it’s a fact that it is so.\nKN: These days it’s very hard to have very deep knowledge about everything. In my opinion, it’s almost impossible, because there’s too much. Even if you look at let’s say some data flow in the modern technology, in the modern data architecture there are too many technologies that can be engaged from the beginning to the end, from end to end solutions.\nMW: And before you had two or three.\nKN: Yes, SQL Server and maybe SSIS and that’s it, right?\nMW: Yeah, and you know that either you use C# applications or SSIS and which is best in this case. Now we have 50 different things and maybe there’s some that I don’t even know that is better than the things I know.\nKN: And everything is changing very quickly.\nMW: Like we said in Azure, the APIs and the demos break all the time.\nKN: Exactly. So if you look at the market, we can see that there are many positions that are available on the market. The market needs more and more people working in IT. So basically the market needs more people, which means young people as well. So what kind of hints would you give to young people if they want to start working in the IT market? Doesn’t matter which technology. It might be Microsoft, might be something else. What kind of advice would you give them, how to start doing this easier?\nMW: I would say there is no way to start easy. If you want to be good, if you want to be good in the market, you must know how things work and why they work. It isn’t enough to be very good and very quick with your mouse and the user interface. It’s when you know why you’re doing this and why it works in a certain way, and how the result is coming. That’s when you are good at your job. So I tell everybody: when you want to be good, you need to go down, you need to drill down into the technology and see why it works and how it works.\nKN: So you mean practice yourself doing this?\nMW: Yeah, see what happens, experiment, do something that you know won’t work and then try to understand why this didn’t work.\nKN: What can help more, in your opinion, some conferences or maybe some online documentation or maybe some webcasts? What helps better to understand the technology?\nMW: A combination. I would say a conference is very good, like SQLSaturday that is free, to see “OK, I’m interested in this topic, I think this technology looks cool”. But then, going further with online documentation, doing labs, doing experiments with big data sets. That’s what makes you better than everybody else. But you must choose first from the smorgasbord of things you can do. And there a conference is the best place because you have sessions about everything.\nKN: And you have access to the experts from their area. As you mentioned, those events are free and you can ask any questions that you need.\nMW: Yeah, and most speakers stay for the entire day and just walk around the people and ask questions and answer questions.\nKN: What is your recipe to be up-to-date with technology these days?\nMW: I have my biggest interest as my job. So sometimes when we sit together, me and my wife on a Friday night, she’s wondering: “do you want to look at TV or are you gonna play with your SQL Server still?” and the answer is most often “I want to play with my SQL Server”. So it needs time. And if you’re interested in it, if you find a passion for things, then you can spend time with it.\nKN: Was it you yesterday when we were waiting for the ferry? Did someone ask about movies?\nMW: Yeah.\nKN: Could you tell us about it?\nMW: Yes, when I work for a week and I’m away, I usually relax playing with the SQL Server or maybe looking at the movie together with my wife. And I’ve told her that movies Friday night she can pick because I don’t remember, I don’t know what we see, because I’m relaxing my brain, emptying the cache. So she did an experiment once that on Saturday she said “can we watch this movie? I’ve heard it’s good”, and I say “yes, sure”. And then about an hour into the movie, I told her “I think I’ve seen this”, and then she smacked me on the head and said, “yes, we saw it yesterday”. So, yeah.\nKN: After one hour, right?\nMW: Yeah, after one hour I realized I saw it less than 24 hours ago. So yeah, movies are not the great thing for information input to me. It’s a time-spending thing.\nKN: I had the same. I can watch a movie or read some books and after half a year I don’t remember what it was about or I don’t remember the actors. Obviously with some exceptions but generally it’s like this. What kind of technology are you working with right now? What is closest to you?\nMW: I try to specialize since SQL Server is so extremely wide now. I try to focus on Data Platform, only SQL Server engine, and Data In, Data Out. So that means I’m doing loading of data warehouses, SSIS, I’m doing performance tuning, also backup restores, often performance issues.\nKN: So mainly on-premises?\nMW: Yes, on-prem and also platform evaluation, you can say, to see if the customer has what it takes to go to SQL 17 or 19, or go to Azure. So I do some kind of inventory to see what they are using and if they can upgrade.\nKN: The new version of SQL Server, 2019, might be very interesting for your customers, or generally for all of us, because it brings some technology that we have never seen with a SQL engine before, yeah?\nMW: Both that, for the GDPR and the city governments companies, but also the performance. We have a lot of customers with a legacy code that has been running since Cursors was a good thing and temp tables were even better. And they’re using a modern approach that is table variables, and the performance increase in 2019 of those system objects and everything would be probably a very good thing for our customers.\nKN: Yeah, exactly. What do you think about Java, Python, R in the SQL Server engine? Maybe not directly in the engine, but\u0026hellip;\nMW: I think it’s cool. It’s good that we can use the data without moving it so far. It’s still in the same machine and you can use their business standard code languages. However, having said that, that’s one of the areas that I decided “that’s not my focus”. So I’m leaving that for other consultants. And I’m kind of happy with that. I have colleagues in SolidQ that are experts in both Python and R services, so we have a good combination. I realize that one man cannot learn everything. Even if it’s hard to accept.\nKN: Yes, unfortunately, it is. What do you think would be the future of ETL, ELT processing? Or what would be the future of SSIS for example in that kind of context?\nMW: The future of SSIS would be that it has components that have wizards or templates with AI-ish things that do a ton of work under the hood. So it sets up machine learning, algorithms, testing things out. And yes, I know there are in Azure already. The machine learning pipeline, the automatic learning classification things, where you learn your data, and I think they will grow and develop in Azure and come down to SSIS. And the features of SSIS, the tight integration, would go up to Azure. So I think more Azure or on-prem would be a moot point. That depends on if you have your machines on-prem or if somebody else, like Azure, has your machines. But you would have the same functionality, same flexibility everywhere. At least that’s my dream. That would be perfect.\nKN: What kind of things the customers should understand if they want or consider moving to Azure for example or to the cloud in general?\nMW: That they shouldn’t move their easiest data load to Azure just because they can. They should do a thorough analysis to see which data load is the best to have in Azure because Azure is not good at everything. There are things that you need to have on-prem, still. So they should do better background work on what to move because moving things back might cost you some money and some time.\nKN: Maybe not all the existing solutions are good enough to move to Azure.\nMW: No, when you have transaction-intensive applications that the client is running on-prem, your application will probably die from moving the server storage up to Azure. So you have to redesign the architecture and have the client in Azure, then it could be done. But are you prepared to do that, are you willing or is it allowed even to do that?\nKN: So you mean that it would definitely be easier to create a proper new solution in the cloud?\nMW: Yes, in some cases it’s better to rewrite an entire system for Azure then using the on-prem version that has a lot of legacy and timing issues.\nKN: But if we are not thinking about existing systems, if we are thinking about new projects, would you recommend to the customer that they use the cloud or maybe still sometimes on-prem solutions, for new projects?\nMW: I would recommend the thing that is best. And sometimes it’s still on-prem that is best for their solutions, and sometimes it’s Azure. And I can’t say a percentage of how many per cent is best in Azure, but it’s very depending on the project and the customer. Because different customers have different maturity in cloud things as well.\nKN: Cool. Thanks, Mikael. At the end of our conversation, could you tell us where we can find you on the Internet or how we can contact you?\nMW: I have a Twitter account that I’m using maybe two days every year, and that is those two days before SQLSaturday because speakers communicate with that. I haven’t really found how to use it yet. And normally email is very good.\nKN: Thank you very much for this conversation.\nMW: Thank you.\nUseful links Mikael\u0026rsquo;s profiles: Twitter | LinkedIn\nMikael\u0026rsquo;s blog: Blog\nRelated events: SQL Saturday Gothenburg\n","date":"2019-11-29T01:11:04Z","image":"/uploads/2019/11/ASF027-SQLSaturday-with-Mikael-Wedham-1200.jpg","permalink":"/2019/11/asf-027-mikael-wedham-interview/","title":"ASF 027: Mikael Wedham interview"},{"content":"In this short post, I will show you step by step on how to install two different modes of a Power BI Data Gateway.\nWhat is it this Data Gateway?\nLet\u0026rsquo;s start from the beginning: What exactly is the Data Gateway?\nData Gateway works just like a bridge that creates a connection between on-premises data (who are not in the cloud) and Power BI cloud-based service. You can install Data Gateway in any servers in the local domain.\nWe have different two types of Data Gateway:\nPersonal Mode On-Premises (also called as Enterprise Mode) You can install both modes of gateway for free. Below, I explain these two different modes of Data Gateway and I will show you the installation process.\nOK, let\u0026rsquo;s do something\u0026hellip;\nInstallation Before you start the installation you should think about which type of Data Gateway you want. Recommended is On-premises Mode because it gives you more capabilities than the personal mode.\nMinimum Requirements .NET 4.6 Framework 64-bit version of Windows 7/ Windows Server 2008 R2 (or later) Recommended 8 Core CPU 8 GB Memory 64-bit version of Windows 2012 R2 (or later) Although we have two different modes of a gateway - there is only one file to be downloaded, from here:\nhttps://powerbi.microsoft.com/en-us/gateway/\nDownloading Power BI Gateway Download the file (gateway) and save file:\nPower BI Gateway Installer When you run the downloaded file - you can choose which gateway you want to install:\nType of data gateway Just like I wrote above, we have two types of Data Gateway Mode.\nPower BI Gateway: On-premises Mode This On-premises was created for a multi-developer environment. Here multiple people can work on the same Data Gateway if the administrator authorized them before. With this Data Gateway, you will have much more fun than Personal Mode. Why? Because of on-premises support not only Power BI just like his brother Personal Mode. Here you can provide quick and secure data transfer between data which is not in the cloud and Microsoft cloud services: PowerApps, Microsoft Flow, Azure Analysis Services, and Logic Apps and of course Power BI. Depending on your needs, you can choose one of them.\nHow to install On-premises Data Gateway In the first step choose On-premises:\nOn-premises data gateway After this, you will see the installation process ongoing:\nInstallation process When installation it\u0026rsquo;s almost done you must register your gateway. The first step to do this is to input your email:\nI named my gateway \u0026ldquo;pola-gate\u0026rdquo;, you can name it whatever you want.\nRegister the gateway In the second step, you must name your gateway. I\u0026rsquo;m naming it \u0026ldquo;pola-gate\u0026rdquo;.\nAfter that, you must create your (name it) \u0026lsquo;Recovery key\u0026rsquo;. icon-bell The recovery key is so important. For example:\nif you change you decide to uninstall the gateway the recovery key will help you avoid other problems with changing all connections. The second situation where you need the Recovery key will be if you want to move your current gateway form one machine to another without changing connections. So, just like with your password, keep your Recovery key in a safe place.\nWhen the installation process is finish - you will see green tick with a message that the gateway is online and ready to go:\nThe gateway is ready to be used So, we have our shiny-new Power BI Gateway up and running on-premises. What\u0026rsquo;s next?\nPower BI service: Where is my gateway? OK, this type of gateway you can find in your Power BI services.\nIf you go to settings panel in Power BI services you will see Manage gateways.\nManage gateways When you click \u0026lsquo;Manage gateways\u0026rsquo; you will see the gateway which we have installed as ready to use.\nGateway Cluster Settings Power BI Gateway: Personal Mode The personal mode can be used only by one business analyst person. This gateway is only for Power BI – cannot be used for other applications just like On-premises. Of course in this type, you can share your Power BI reports with other users but you will be the only one who has administration access. Installing this type is easy-peasy.\nHow to install the Personal Mode In this case, the installation process is very similar just like On-premises. Here you don\u0026rsquo;t need the gateway\u0026rsquo;s name and you don\u0026rsquo;t need the recovery key. It\u0026rsquo;s a small thing that you should know it can not be moved to any other machine.\nFirst, choose the personal mode before you click Next:\nHow to install personal mode data gateway Process installing on-premises data gateway Adding e-mail address Add your Email address. After this, you will see:\nData gateway personal mode ready to use Congratulations! Your Personal Mode Gateway is ready to use.\n*The one thing that you should know is that you will not see the personal mode in Power BI services just like On-premises.\nPower BI Services Once you installed the gateway on a computer, it will be nice to complete this job by adding a data source to the gateway.\nWhy? The answer is simple. Because the Data Gateway is the only bridge between the cloud and data sources in your domain (enterprise). So, you need to build this bridge by letting Power BI service know where is a particular server and database located in your local network and how to connect to it.\nHence, you should add new data sources to your available gateway. You can add single or multiple data sources.\nAdding data sources Here you can find the list of available data sources type https://docs.microsoft.com/en-us/power-bi/service-gateway-data-sources#list-of-available-data-source-types\nNow, when you log in to Power BI services click the gear icon. Then click Manage gateways:\nAdding sources to Power BI Services After this step, you can add data sources. If you want to add multiple data sources you must repeat this process for every single data source.\nSteps how to add sources to use gateway In this step you must complete the required information:\nData Source Name Data Source Type Server Database Authentication Method The last step to add sources in data gateway in Power BI services The server and Database must be the same as your data source setting in the Power BI report.\nClick Add button, and if you see Connection Successful the process succeeds. Now you can use these data sources in your Power BI.\nConnection Successful Remove data sources If you decide to remove your data sources from Gateway in Power BI Service, back to data sources and there click the ellipsis and Remove.\nHow to remove sources form data gateway icon-star Remember: when you remove data sources, any dashboards, and reports whose are based on these sources will be broken. How to remove Data Gateway from Power BI Services? When you decide to not use them anymore you can do this in a very easy way.\nGo to Power BI Services -\u0026gt; Settings -\u0026gt; Manage Gateways - in this place select the one you do not need anymore and click the ellipsis and click Remove.\nHow to remove data gateway from Power BI Service After this, you can see communicate:\nHow to remove Gateway from your computer? Go to the Control Panel in your computer, there find Programs and Features:\nRemoving data gateway from computer Then you must find an application that you want to remove, in this case, it is \u0026ldquo;On-premises data gateway\u0026rdquo;. Afterwards, you must tick and select the application and click the \u0026ldquo;Uninstall\u0026rdquo; button to begin the uninstallation process.\nLast step uninstalling on-premises data gateway from computer Your Gateway will be removed from your computer and you will be able to reinstall the application if needed.\nNow, I hope that you know a little bit more about Data Gateway in Power BI. What is it? Why we can need them? How to install both modes of Data Gateways and which one is more suitable for you? So, if somebody asks you during an interview about Data Gateway in Power BI, you would easily explain what it is.\nSee you in the next post.\n","date":"2019-11-14T07:32:57Z","image":"/uploads/2019/11/volcanic-3442300_1200.jpg","permalink":"/2019/11/installation-and-setting-up-power-bi-gateway/","title":"Installation and setting up Power BI Gateway"},{"content":"Hello all! An extraordinary week has just passed. Two big conferences took place in the USA, PASS Summit and Ignite, where Microsoft announced plenty of new products or updates. Keynotes and sessions from Ignite (most of them) can be seen on the website. Take a look at what I prepared for you down below here.\nPress SQL Server 2019 is now generally available\nThis version is a game-changing release for Microsoft and the industry.\nModern Datawarehousing Using Azure Data Lake, CDM Folders And Power BI Dataflows\nDave Ruijter explains where is the place for Power BI dataflows in the modern approach to warehousing.\nWhat’s new in Azure Cosmos DB: November 2019\nMany interesting features in this update and I believe that AutoPilot (preview) was the most anticipated one.\nMicrosoft Ignite 2019 Announcements – The Ignite Book of News\nPaul sharing the e-book of announcements that you must go through.\nSimply unmatched, truly limitless: Announcing Azure Synapse Analytics\nEnd-to-end experience of building and deploying rich analytics scenarios and how you can automatically generate predictive models.\nPerformance tips for Azure Cosmos DB and .NET\nCheck this out how to test performance and measure the costs.\nNew in Azure Synapse Analytics: CICD for SQL Analytics using SQL Server Data Tools\nThat\u0026rsquo;s a long-awaited feature: #SSDT for #AzureSynapseAnalytics (formerly known as Azure SQL Data Warehouse).\nAzure Synapse Analytics combines data warehouse, lake and pipelines\nThe third generation of SQL DW, along with new capabilities in preview.\nThe future of content lifecycle management in Power BI (Video)\nDevOps for PowerBI? Yes! Why not!? Finally, we have a deployment pipeline. Define rules for slots (environments) and promote changes with one-click. Fabulous!\nDevice Offline detection with Durable Entities\nGreat read on device offline detection with Durable Entities by Kees Schollaart.\nThe November 2019 release of Azure Data Studio is now available\nThe new release includes SQL Server 2019 support, PowerShell notebooks, Jupyter Books, Schema Compare and more\u0026hellip;\nLarge models in Power BI Premium public preview\n400 GB instead of 10GB? Wow!\nA word from Microsoft’s CEO Mixed reality is one of the defining technologies of our time – and we are bringing this new medium to more customers across every industry with the GA of HoloLens 2 and rollout of Minecraft Earth, beginning in the UK. https://t.co/M4ZfXMQ8S6\n— Satya Nadella (@satyanadella) November 7, 2019 Video: Azure Synapse Analytics - Next-gen Azure SQL Data Warehouse https://www.youtube.com/watch?v=tMYOi5E14eU\nSmile Corner ","date":"2019-11-10T17:55:08Z","image":"/uploads/2019/11/news-tyler-nix-tPz-uoz5RBU-unsplash-1200.jpg","permalink":"/2019/11/last-week-reading-2019-11-10/","title":"Last Week Reading (2019-11-10)"},{"content":"This week I\u0026rsquo;m offering you many posts about Cosmos DB, Azure Data Factory and how to design reliable Azure applications.\nHave a great week also for those of you who are not at Microsoft Ignite or PASS Summit.\nPress How to Think Like the SQL Server Engine: What’s a Key Lookup?\nOne of the episodes of awesome series from Brent Ozar.\nConfigure Self-Hosted IR as a proxy for Azure-SSIS IR in ADF\nStep by step guide if you want to access data on-premises without joining your Azure-SSIS IR to a virtual network.\nADF Adds Avro and CosmosDB Data Flow Connectors\nAggregate and join data on the fly from CosmosDB.\nAzure Cosmos DB – Programatically Connect to a preferred location using the SQL API\nSet up the location from your app.\nAzure Cosmos DB: REST API Reference\nWatch this video to learn how to quickly fiddle with Postman to work with RestAPI and CosmosDB. Plus a list of supported REST API Versions.\nExport data from Cosmos DB to SQL Server\nArjun describes how to export documents to JSON and import all of them to SQL Server.\nPassing a file from an Azure Logic App to a Web API\nSometimes you need a small trick.\nBook: Praktyczne uczenie maszynowe\nMy polish friend, Marcin Szeliga, presents his new book about Machine Learning. Sorry, polish language only.\nResiliency checklist for specific Azure services\nYou must go through the list\u0026hellip;\nResiliency patterns\n\u0026hellip; and these patterns\u0026hellip;\nDesigning reliable Azure applications\n\u0026hellip; if you care of resilience of the designed system.\nASF 026: Aaron Bertrand interview\nJust to remind you, if you not aware yet, there is a fresh new episode of \u0026ldquo;Ask SQL Family\u0026rdquo;.\nPower BI End-To-End Features\nMelissa Coates updated her amazing diagram. Do you know it?\nA word from Microsoft’s CEO As we wrap up #NDEAM, I wanted to share another example of an innovation built to empower everyone. Eyes First games are designed for people with mobility and speech disabilities to be able to play games using just their eyes. https://t.co/6ulFTpeoCl\n— Satya Nadella (@satyanadella) October 30, 2019 Video: Introducing Data Virtualization https://www.youtube.com/watch?v=oJTCKvwsyo8\nSmile Corner ","date":"2019-11-03T23:59:14Z","image":"/uploads/2019/09/news-alex-Yvl081TVnvA-unsplash-1200.jpg","permalink":"/2019/11/last-week-reading-2019-11-03/","title":"Last Week Reading (2019-11-03)"},{"content":"This post is a part of the series about SSDT and database project leveraging Visual Studio.\nWe already know that SSDT can cover all objects but only on a database level, such as:\ntables, view, stored procedures, functions, user types, assemblies, etc. We can not manage out of the box with SSDT is everything behind the scope, e.g.:\nSQL Logins, SQL Jobs, Extended Events. There is another category of thing, which exist on a database level, but can\u0026rsquo;t be managed by SSDT natively.\nNamely: The DATA.\nIt\u0026rsquo;s a very common scenario when we have a database (project) and require some data in selected tables even in the freshly deployed, empty database.\nNomenclature for these data/tables that you can meet are:\n- reference data\n- master data\n- dictionary\nWhich term have you heard and is closest to you?\nBy default, SSDT does not support scripting or deploying the data.\nHence: the question is: how we can cope with that?\nThe answer is the Post-Deployment script.\nHaving said that - you already might know than the game is just beginning\u0026hellip;\nPost Deployment There are several ways how to approach this problem.\nI will show you some solutions I have used for projects I\u0026rsquo;ve worked for to date.\nI would divide the problem into two things:\n1) How to script the existing data\n2) How to build a deployment of the reference data, considering the following aspects:\na) Inserting new rows\nb) Update existing data (in target table)\nc) Should I delete rows that don\u0026rsquo;t exist in the source?\nLet\u0026rsquo;s take a look (agenda):\nType of reference data Scenario #1 (insert new rows only) This is probably the most common way to generate reference data.\nIt is not fast, but it can work for small tables. Also, any developer can write it, not just database developers.\nTarget is not empty, but new rows can come from the code and need to be added.\nIF NOT EXISTS (SELECT 1 FROM [Person].[PhoneNumberType] WHERE [PhoneNumberTypeID] = 1) BEGIN INSERT INTO [Person].[PhoneNumberType] ([PhoneNumberTypeID], [Name], [ModifiedDate]) VALUES (1, \u0026#39;Cell\u0026#39;, \u0026#39;20171213 13:19:22.273\u0026#39;) END --Repeat above block for every row in a table. Scenario #2 (insert only) - initial load data In this scenario, we expect to insert reference data rows only once - for the first time after the creation of a database and table. The table is empty, so that indicates whether inserting the data or not. Once done - we don\u0026rsquo;t have to touch the data as we know that the data is static in there and no one gonna changes it.\nIF NOT EXISTS (SELECT TOP (1) * FROM [Person].[PhoneNumberType]) BEGIN ;WITH cte_data as (SELECT [PhoneNumberTypeID], [Name], [ModifiedDate] FROM (VALUES (1,\t\u0026#39;Cell\u0026#39;,\t\u0026#39;20171213 13:19:22.273\u0026#39;) , (2,\t\u0026#39;Home\u0026#39;,\t\u0026#39;20171213 13:19:22.273\u0026#39;) , (3,\t\u0026#39;Work\u0026#39;,\t\u0026#39;20171213 13:19:22.273\u0026#39;) ) as v ([PhoneNumberTypeID], [Name], [ModifiedDate]) ) INSERT INTO [Person].[PhoneNumberType] ([PhoneNumberTypeID], [Name], [ModifiedDate]) SELECT [PhoneNumberTypeID], [Name], [ModifiedDate] FROM cte_data; END A drawback of this approach: data could change accidentally by developers, external system (when should not) or undiscovered process. That\u0026rsquo;s why is better to protect these data and redeploy and overwrite all rows over and over again.\nMERGE all things\nWhat if you must \u0026ldquo;simply\u0026rdquo; reflect all you have in the source (code repository), updating whatever exists or not in the target database tables?\nThe things are getting complicated, right?\nA MERGE statement can come with help then.\nHave a look at these 2 scenarios:\nScenario #3 (Full update, including Delete) By creating a MERGE statement we do ensure that the data is the same as in repo. Not only the new rows but also all unexpected rows which must be deleted, or changes made for existing rows which must be updated and return to the desired state. The script should do all actions at once:\nINSERT - for these rows which are in the source but not in the target yet UPDATE - for all existing rows DELETE - for all rows which present in the target but not in the source Let\u0026rsquo;s create a code - MERGE statement for the table. Now, you will see the code created by one tool that I introduce later:\nCREATE PROCEDURE [data].[Populate_Person_PhoneNumberType] AS BEGIN /* Table\u0026#39;s data: [Person].[PhoneNumberType] Data Source: [DEV19].[AdventureWorks2014] Created on: 18/10/2019 15:06:49 Scripted by: DEV19\\Administrator Generated by Data Script Writer - ver. 2.0.0.0 */ PRINT \u0026#39;Populating data into [Person].[PhoneNumberType]\u0026#39;; IF OBJECT_ID(\u0026#39;tempdb.dbo.#Person_PhoneNumberType\u0026#39;) IS NOT NULL DROP TABLE #Person_PhoneNumberType; SELECT * INTO #Person_PhoneNumberType FROM [Person].[PhoneNumberType] WHERE 0=1; SET IDENTITY_INSERT #Person_PhoneNumberType ON; INSERT INTO #Person_PhoneNumberType ([PhoneNumberTypeID], [Name], [ModifiedDate]) SELECT CAST([PhoneNumberTypeID] AS int) AS [PhoneNumberTypeID], [Name], [ModifiedDate] FROM (VALUES (1,\t\u0026#39;Cell\u0026#39;,\t\u0026#39;20171213 13:19:22.273\u0026#39;) , (2,\t\u0026#39;Home\u0026#39;,\t\u0026#39;20171213 13:19:22.273\u0026#39;) , (3,\t\u0026#39;Work\u0026#39;,\t\u0026#39;20171213 13:19:22.273\u0026#39;) ) as v ([PhoneNumberTypeID], [Name], [ModifiedDate]); SET IDENTITY_INSERT #Person_PhoneNumberType OFF; SET IDENTITY_INSERT [Person].[PhoneNumberType] ON; WITH cte_data as (SELECT CAST([PhoneNumberTypeID] AS int) AS [PhoneNumberTypeID], [Name], [ModifiedDate] FROM [#Person_PhoneNumberType]) MERGE [Person].[PhoneNumberType] as t USING cte_data as s ON t.[PhoneNumberTypeID] = s.[PhoneNumberTypeID] WHEN NOT MATCHED BY target THEN INSERT ([PhoneNumberTypeID], [Name], [ModifiedDate]) VALUES (s.[PhoneNumberTypeID], s.[Name], s.[ModifiedDate]) WHEN MATCHED THEN UPDATE SET [Name] = s.[Name], [ModifiedDate] = s.[ModifiedDate] WHEN NOT MATCHED BY source THEN DELETE ; SET IDENTITY_INSERT [Person].[PhoneNumberType] OFF; DROP TABLE #Person_PhoneNumberType; -- End data of table: [Person].[PhoneNumberType] -- END GO Scenario #4 (Full update, excluding Delete) For Scenario 4 you must just omit lines 37-38 in order not to delete rows (added by someone manually in the target database, but that are not present in the source).\nCREATE PROCEDURE [data].[Populate_Person_PhoneNumberType] AS BEGIN /* Table\u0026#39;s data: [Person].[PhoneNumberType] Data Source: [DEV19].[AdventureWorks2014] Created on: 18/10/2019 15:06:49 Scripted by: DEV19\\Administrator Generated by Data Script Writer - ver. 2.0.0.0 */ PRINT \u0026#39;Populating data into [Person].[PhoneNumberType]\u0026#39;; IF OBJECT_ID(\u0026#39;tempdb.dbo.#Person_PhoneNumberType\u0026#39;) IS NOT NULL DROP TABLE #Person_PhoneNumberType; SELECT * INTO #Person_PhoneNumberType FROM [Person].[PhoneNumberType] WHERE 0=1; SET IDENTITY_INSERT #Person_PhoneNumberType ON; INSERT INTO #Person_PhoneNumberType ([PhoneNumberTypeID], [Name], [ModifiedDate]) SELECT CAST([PhoneNumberTypeID] AS int) AS [PhoneNumberTypeID], [Name], [ModifiedDate] FROM (VALUES (1,\t\u0026#39;Cell\u0026#39;,\t\u0026#39;20171213 13:19:22.273\u0026#39;) , (2,\t\u0026#39;Home\u0026#39;,\t\u0026#39;20171213 13:19:22.273\u0026#39;) , (3,\t\u0026#39;Work\u0026#39;,\t\u0026#39;20171213 13:19:22.273\u0026#39;) ) as v ([PhoneNumberTypeID], [Name], [ModifiedDate]); SET IDENTITY_INSERT #Person_PhoneNumberType OFF; SET IDENTITY_INSERT [Person].[PhoneNumberType] ON; WITH cte_data as (SELECT CAST([PhoneNumberTypeID] AS int) AS [PhoneNumberTypeID], [Name], [ModifiedDate] FROM [#Person_PhoneNumberType]) MERGE [Person].[PhoneNumberType] as t USING cte_data as s ON t.[PhoneNumberTypeID] = s.[PhoneNumberTypeID] WHEN NOT MATCHED BY target THEN INSERT ([PhoneNumberTypeID], [Name], [ModifiedDate]) VALUES (s.[PhoneNumberTypeID], s.[Name], s.[ModifiedDate]) WHEN MATCHED THEN UPDATE SET [Name] = s.[Name], [ModifiedDate] = s.[ModifiedDate] ; SET IDENTITY_INSERT [Person].[PhoneNumberType] OFF; DROP TABLE #Person_PhoneNumberType; -- End data of table: [Person].[PhoneNumberType] -- END GO Cool. Merge statement is not easy to remember and also adopting it for dozens of tables is a time-consuming process. Furthermore, to standardise the code for all tables - would be perfect to use a tool to script the data.\nThankfully, you are not alone here. You have two options.\nScripting master data In this post, I want to show you two tools which can help:\nData Script Writer (Desktop Application for Windows) Generate SQL Merge (T-SQL Stored Procedure) Data Script Writer (desktop app) This is open-source, simple Desktop application for Windows which allows you to connect to your database, list of tables, select the ones you are interested in and with one button - script ALL the data to files.\nLet\u0026rsquo;s familiar with the process:\nRun the app and connect to your server and select database: Data Script Writer (Desktop App) - Connect to Server Once hit CONNECT the App will load the list of all tables with information of the number of rows per each table. Data Script Writer - select tables and script method Check all tables you want to script. Optionally you can select a method of pushing the data into the target: a) INSERT (see: scenario 2 above) b) MERGE (see: scenario 3 above)\nc) MERGE without DELETE (see: scenario 4 above)\nd) MERGE NEW ONLY (see: scenario 1 above) 5. Click \u0026ldquo;Generate script\u0026rdquo; to do the things.\nThe last step is to put all these scripts into our SSDT database project.\n\u0026ldquo;Show All Files\u0026rdquo; button To do this - follow the steps:\nCopy these scripts to the location of the database project. (e.g.: Data/Stored Procedure) Select the database node. Switch on the option \u0026ldquo;Show all files\u0026rdquo; to be able to see hidden files (not attached to the project) Select all required files and click \u0026quot;\u0026quot; Check everything is ok by building project The application is available to download here (git repo): Data Script Writer.\nGenerate SQL Merge (script) It\u0026rsquo;s an open-source script that helps you generate MERGE statement with static data.\nFirstly, you must install the script on the server. Installation is trivial:\nSimply execute the script, which will install it in \u0026ldquo;master\u0026rdquo; database as a system procedure (making it executable within user databases).\nAs I\u0026rsquo;m a PowerShell enthusiast, I have created a simple script that does this for you:\n$url = \u0026#39;https://raw.githubusercontent.com/readyroll/generate-sql-merge/master/master.dbo.sp_generate_merge.sql\u0026#39;; $temp = [System.IO.Path]::GetTempPath() $outputfile = \u0026#34;$temp\\sp_generate_merge.sql\u0026#34; Write-Host \u0026#34;Downloading script $outputfile ...\u0026#34; Invoke-WebRequest -Uri $url -OutFile $outputfile $script = Get-Content $outputfile -Raw # Exec script $databaseServer = \u0026#39;localhost\u0026#39; $server = New-Object Microsoft.SqlServer.Management.Smo.Server($databaseServer) $server.ConnectionContext.ConnectionString = $connectionString $sql = [IO.File]::ReadAllText($outputfile) $batches = $sql -split \u0026#34;GO\\r\\n\u0026#34; $VerbosePreference = \u0026#39;Continue\u0026#39; Write-Host \u0026#34;Batches prepared to be executed: $($batches.Count)\u0026#34; $server.ConnectionContext.BeginTransaction() $batches | ForEach-Object { try { $r = $server.ConnectionContext.ExecuteNonQuery($_) } catch { $server.ConnectionContext.RollBackTransaction() Write-Warning -Message $_ throw } } if ($server.ConnectionContext.TransactionDepth -gt 0) { $server.ConnectionContext.CommitTransaction() Write-Host \u0026#34;Instalattion completed successfully.\u0026#34; } It\u0026rsquo;s worth to mention that the script can be stored in [master] database to be used from any user database context.\n--Mark the proc as a system object to allow it to be called transparently from other databases EXEC sp_MS_marksystemobject sp_generate_merge GO How to use it once installed? Let\u0026rsquo;s take the table [PhoneNumberType] from [AdventureWorks] database as an example:\nEXEC AdventureWorks.dbo.sp_generate_merge @schema = \u0026#39;Person\u0026#39;, @table_name =\u0026#39;AddressType\u0026#39;; As a result, you\u0026rsquo;ll get the following script:\nUSE [AdventureWorks2014] GO --MERGE generated by \u0026#39;sp_generate_merge\u0026#39; stored procedure --Originally by Vyas (http://vyaskn.tripod.com/code): sp_generate_inserts (build 22) --Adapted for SQL Server 2008+ by Daniel Nolan (https://twitter.com/dnlnln) SET NOCOUNT ON SET IDENTITY_INSERT [Person].[PhoneNumberType] ON MERGE INTO [Person].[PhoneNumberType] AS [Target] USING (VALUES (1,N\u0026#39;Cell\u0026#39;,\u0026#39;2017-12-13T13:19:22.273\u0026#39;) ,(2,N\u0026#39;Home\u0026#39;,\u0026#39;2017-12-13T13:19:22.273\u0026#39;) ,(3,N\u0026#39;Work\u0026#39;,\u0026#39;2017-12-13T13:19:22.273\u0026#39;) ) AS [Source] ([PhoneNumberTypeID],[Name],[ModifiedDate]) ON ([Target].[PhoneNumberTypeID] = [Source].[PhoneNumberTypeID]) WHEN MATCHED AND ( NULLIF([Source].[Name], [Target].[Name]) IS NOT NULL OR NULLIF([Target].[Name], [Source].[Name]) IS NOT NULL OR NULLIF([Source].[ModifiedDate], [Target].[ModifiedDate]) IS NOT NULL OR NULLIF([Target].[ModifiedDate], [Source].[ModifiedDate]) IS NOT NULL) THEN UPDATE SET [Target].[Name] = [Source].[Name], [Target].[ModifiedDate] = [Source].[ModifiedDate] WHEN NOT MATCHED BY TARGET THEN INSERT([PhoneNumberTypeID],[Name],[ModifiedDate]) VALUES([Source].[PhoneNumberTypeID],[Source].[Name],[Source].[ModifiedDate]) WHEN NOT MATCHED BY SOURCE THEN DELETE; DECLARE @mergeError int , @mergeCount int SELECT @mergeError = @@ERROR, @mergeCount = @@ROWCOUNT IF @mergeError != 0 BEGIN PRINT \u0026#39;ERROR OCCURRED IN MERGE FOR [Person].[PhoneNumberType]. Rows affected: \u0026#39; + CAST(@mergeCount AS VARCHAR(100)); -- SQL should always return zero rows affected END ELSE BEGIN PRINT \u0026#39;[Person].[PhoneNumberType] rows affected by MERGE: \u0026#39; + CAST(@mergeCount AS VARCHAR(100)); END GO SET IDENTITY_INSERT [Person].[PhoneNumberType] OFF SET NOCOUNT OFF GO There is more option to be chosen, but it\u0026rsquo;s not my goal to present you all of them. One the website you find out all details.\nWhen updating the tables? The best moment to do that is just after deployment the changes. That means: post-deployment script.\nThe first, easiest way, is to only append the script file to the deployment script:\n:r ..\\Data\\Person.PhoneNumberType.sql Required property of Post-Deployment file A drawback of that approach is that the whole script appears in there whenever deploying the database. And even worse: the whole script won\u0026rsquo;t be parsed like all other scripts which are being defined as post-deployment.\nMy recommended method I would like to suggest one particular approach to address that case. I have applied it many times in multiple projects and it just works.\nLet\u0026rsquo;s take a look at the code. We require:\nSeparate schema (e.g. \u0026ldquo;data\u0026rdquo;) - optionally Script inserting the data wrapped up in Stored Procedure One \u0026ldquo;principal\u0026rdquo; stored procedure which calls all others. Post-deployment script when you have only one line of code - execute the \u0026ldquo;principal\u0026rdquo; procedure Does that make sense? I hope so. To make sure that you understand it better - have a look the following pic:\nAll you need is\u0026hellip; these files To convince you more to this approach, let me list all its advantages:\nThe stored procedures are part of the database project. They will be validated and compiled, thus you avoid potential errors with uncontrolled code The changes to the SP appears in output script only when something changes. It\u0026rsquo;s easier to review the script before a run as it doesn\u0026rsquo;t contain unnecessary code Having separate schema - you can control who has access to the SPs to protect against accidental executions Having \u0026ldquo;principal\u0026rdquo; SP - you can control what and when will be executed, means populated. In Post-deployment script - there is only one line of code which never is changed. It is important as this script is not built, so you must trust that it contains faultless code. For me - it\u0026rsquo;s much more than enough reasons to apply this methodology.\nicon-github I populated a small table with data in this post. If you want to see entire code - check that repo on GitHub: ssdt-demo\nSummary Let\u0026rsquo;s wrap up what we learned today. I do distinguish a few scenarios of updating the data in the target server. Perhaps not all of them is necessary for your scenario, but it works for me, that\u0026rsquo;s why I decided to share. The following table allows you to understand them better:\nHow does your solution look like?\nI\u0026rsquo;m not trying to tell you that my solution is the best (even if I think that ;) )\nYou don\u0026rsquo;t have to like it, maybe it just would not work for you, or you have particular requirements. Each project is specific, although 90% of them can use the examples presented above.\nWhat is your experience with this area?\nMaybe you know a better way to do the things?\nDo you have something to add here?\nPlease do and leave a comment.\n","date":"2019-10-30T07:30:50Z","image":"/uploads/2019/10/adi-goldstein-f_0t4fYEauU-unsplash-1200.jpg","permalink":"/2019/10/script-deploy-the-data-for-database-from-ssdt-project/","title":"Script and deploy the data for database from SSDT project"},{"content":"Hello and good morning! We have Tuesday morning, which means I experienced a delay in this post. But finally, it is out and have a look at how much stuff I have for you today. Have a good week!\nPress Simplifying ETL in the Cloud, Microsoft Releases Azure Data Factory Mapping Data Flows\nADF MDF is GA now. Find out what is that and how this does help with your ETL?\nThree ways to leverage composite indexes in Azure Cosmos DB\nWhat you need to know about composite indexes.\nAzure Data Factory Mapping Data Flows for Big Data Lake Aggregations and Transformations\nRon L\u0026rsquo;Esteve explores data cleansing and aggregation features of Mapping Data Flow.\nADF Adds Hierarchical \u0026amp; JSON Data Transformations to Mapping Data Flows\nWith this new feature of Mapping Data Flows you can ingest, transform, generate schemas, build hierarchies, and sink complex data types using JSON in data flows.\nYou, Me, \u0026amp; CI/CD: Deploying Azure Analysis Services Tabular Models through Azure Pipelines\nLiam Dunphy describes his practical experience of building robust deployment process for AAS Tabular Models with Azure DevOps.\nOverview of how to do AI and ML in Power BI Desktop and Service\nNice table summarizing AI \u0026amp; ML capabilities in PowerBI.\nLeverage Azure premium file shares for high availability of data\nWhat does Azure premium file shares offer for SQL Server failover cluster instance?\nHow I learned to stop worrying and love Cosmos DB’s Request Units\nDo understand Request Units and learn how to determine the required throughput.\nMonitor and debug with metrics in Azure Cosmos DB\nFind out how Azure Cosmos DB metrics can be used to analyze and debug common use cases.\nData Lineage and Power BI: Why is it important?\nIn this post, you will find more explanation of what data lineage is and what’s it does.\nData lineage (preview)\nThis also is a good complement post about Data lineage.\nA word from Microsoft’s CEO Improving health outcomes while reducing costs is one of our biggest challenges – and together with @Humana we’re working to address it, applying the power of Azure AI and Microsoft 365 to build new solutions to help people manage their health. https://t.co/eAQI7Jkzae\n— Satya Nadella (@satyanadella) October 21, 2019 Video: Intro to Power BI Administration https://www.youtube.com/watch?v=oxot-ZpKm4Q\nSmile Corner ","date":"2019-10-29T08:03:39Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/10/last-week-reading-2019-10-27/","title":"Last Week Reading (2019-10-27)"},{"content":" Introduction Aaron Bertrand is a Product Manager at SentryOne, with industry experience dating back to Classic ASP and SQL Server 6.5.\nIn his spare time, he is either playing volleyball, curling, blogging at sqlperformance.com, blogs.sentryone.com and sqlblog.org.\nHe is contributing to SQL conversations on twitter and dba.stackexchange.com.\nHas been a Microsoft MVP since 1997-98. He wrote chapters for two charity-driven books, SQL Server MVP Deep Dives Volume 1 and Volume 2 and speaks frequently at conferences, user group meetings, and SQL Saturday events around the world.\nThis talk has taken place during SQLDay 2019 in Wroclaw, Poland on 15 May 2019 (Wednesday).\nInterviewers: Kamil Nowinski (T), Michal Sadowski (T).\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript Kamil Nowinski: Hello, Aaron. Thank you for accepting the invitation to this podcast, Ask SQL Player.\nAaron Bertrand: Sure, no problem.\nKN: So, could you introduce yourself at the beginning?\nAB: Sure! My name is Aaron Bertrand, I’m a Technical Product Manager for SentryOne and I do a lot of community stuff with SQL Server.\nKN: And where do you live?\nAB: I live in Charlotte, North Carolina.\nKN: Are you Canadian?\nAB: I am Canadian, so I spent the first 23 years of my life in Canada and then I moved to the US.\nKN: Which city in Canada?\nAB: North Bay, Ontario, which is about 3 hours north of Toronto.\nKN: What do you do for a living in more detail?\nAB: I manage three teams that work on various products in the SentryOne Software Suite, including SentryOne SQL Sentry is the component of that monitors SQL Server. And then we have other tools that monitor Azure SQL Database and Azure SQL Data Warehouse, VMware, Windows, etc. One of my teams also manages Plan Explorer, which is our free query tuning tool.\nKN: Yeah, Plan Explorer is a great tool. Every time I need to check the explorer plan I use it, especially if the plan is much bigger than the normal few notes.\nAB: Yes, Management Studio is fine for small plans but as soon as you get anything really complicated, it shows its limits pretty quickly.\nMichal Sadowski: How did you start working with SQL Server?\nAB: I was working at a start-up and the first couple of projects that we had were building websites for companies, one of them was a condom company and they needed a database to host their e-commerce, so the list of products, the prices, shopping carts, all that kind of stuff. So we implemented that in SQL Server. It was the first time I touched SQL Server as a database. It was SQL Server 6.5 and that was my first taste of that. We didn’t have a database person but we needed to do database stuff. I was a web developer and I became\u0026hellip;\nKN: So a very common scenario even in those days.\nAB: Yes, I became the database guy because we didn’t have one and I was the one working on the web application code the most.\nMS: What year was this? You mentioned SQL Server 6.5.\nAB: Yeah, this was 1997 or 1998.\nMS: More than 20 years?\nAB: Yes, more than 20 years.\nKN: As experienced.\nAB: All this grey hair is from SQL Server. And kids.\nKN: What topic did you cover during SQLDay this year?\nAB: I had two sessions, one was on new T-SQL features in SQL Server 2016, 2017 and beyond, and the other one was the one I just finished, which was a “T-SQL: Bad Habits and Best Practices” talk.\nKN: I haven’t been to this session. I think I’ve seen your session like three years ago.\nAB: Three years ago I gave the same “Bad Habits” talk.\nKN: Do I remember correctly that you’ve been a big opponent of using MERGE Statement?\nAB: I am very opposed to MERGE in production, yes.\nKN: But, as far as I know, I know there are a few unresolved issues with this in SQL Server’s engine, but it’s very unlikely to reach them in some scenarios. There must be a specific scenario. It still can happen but\u0026hellip;\nAB: Yes, there are specific wrong results bugs which are really bad but the bigger thing is I think that the MERGE Statement gives users this false sense of “this is one operation that is completely self-enclosed and completely isolated from any other transactions that are going on” and that’s not really the case because it is actually implemented under the covers as multiple operations. Even though it’s a single statement, there are multiple things going on under the covers, so without using the right isolation level, either by surrounding the transaction with SERIALIZABLE or using UPDLOCK on the target of the MERGE Statement, you’re still vulnerable to the same race conditions that we’ve always had to deal with. So I feel like by combining these things into one statement you’re masking the fact that you actually do still need these controls and this transaction wrapper around the statement. There are some other weird things that happen in triggers, like triggers will fire multiple times, once for each operation, and that affects @@ROWCOUNT - it will always be equal to the sum of the rows affected, so the example I gave today was you have an operation that deletes 50 rows, right? The MERGE statement ends up deleting 50 rows but the UPDATE trigger fires, the INSERT trigger fires, the DELETE trigger fires, and they all say 50 rows. So if you’re checking for @@ROWCOUNT . and you say: if @@ROWCOUNT. is greater than zero, go do something, all three of those triggers are going to go do that thing.\nKN: Yeah but this scenario is I think more due to the triggers, not like MERGE. With the triggers, you have a lot of problems.\nAB: But triggers don’t do this. If you have a separate UPDATE statement and a separate INSERT statement, those triggers fire independently, and not twice.\nKN: That’s right. But yesterday I had a session about using SSDT to develop a database and how to deploy these things. One of the things that I showed to the audience was how to deploy the data, some reference data table or some dictionary. I gave this example; use the MERGE statement instead of a procedure to deploy your data, your dictionary for example. Then, one of the attendees told me that you don’t use this MERGE and that you suggest not to use MERGE. But I think in that specific scenario it should be fine? Especially when you do this only during the deployment and also you have full control, you can disable some triggers for example or you can switch the mode on the single user if you really need to.\nAB: The thing that I always like to get across when I’m teaching about bad habits is that you can always find some scenario, some exception where it’s okay, but you’re still teaching that thing, and you’re not necessarily giving all the context of “you shouldn’t use it anywhere else, this is the only place where this is safe”. If that’s the only place where that’s safe and the only thing you’re getting out of using MERGE over using separate statements is ten fewer characters, I don’t think that’s worth it, personally. And that’s what I try to get across in this talk is that it’s not so much about you knowing\u0026hellip; Like using shorthand for dates and you say DATEPART(w, date) of a date is a day of the week, not a week. You know that I know that, but if I write code that says that and someone else reads my code, they don’t know that. They don’t have that extra context. And that context gets lost if you just put MERGE statements where you know it’s safe to use, you’re teaching other people that MERGE is safe to use.\nKN: OK, yes, that’s what I wanted to show. That there are also some specific scenarios where you can use it and some exceptions.\nAB: Absolutely, as long as 1) you’re not just teaching that to people and saying “use MERGE everywhere” and 2) you’re the one who controls that piece of code, because if someone else inherits it and then it goes wrong\u0026hellip; “Why did you use MERGE here, because it has this problem or this race condition” or whatever.\nKN: I totally agree. So we are now at SQLDay conference in Wrocław, Poland, 3rd day.\nMS: You have mentioned that you’ve had two sessions already, but there was also a workshop that was scheduled for the morning, but it didn’t happen.\nAB: I know, and I feel bad.\nMS: Could you shed some light on that?\nAB: Yes, so this was mostly my fault but I also want to blame American Airlines a little bit. So, I was supposed to leave from Charlotte at 10 o’clock on Saturday night, which would get me into Heathrow at 11 in the morning on Sunday. And I had a connecting flight at 2 o’clock from Heathrow that would get me to Warsaw at 5:00 p.m., and then Grant landed at the same time. He was scheduled to be on the same flight as me. And we were going to drive here and be here by 8:30-9:00.\nKN: Plenty of time, three hours?\nAB: Plenty of time. Well, my flight in Charlotte, first it was too wet to load the cargo onto the plane, then there was a plane parked in the gate next to us and so we couldn’t back out of the gate because this plane was parked there and it was blocking. I don’t know why they construct gates in a way that one plane can block another, but it was blocking. It took us four and a half hours before we backed out of the gate. Four and a half-hour delay. So I left Charlotte at 2:30 in the morning on Saturday night, which means I got into Heathrow at 3:00 [pm] o’clock in the afternoon London time, which means I missed my two o’clock flight here. The next flight was six hours later on a completely different airline, and because they paid for my ticket while I was in the air, that was how I was getting here. And I could only fly to Warsaw. If I had been able to reschedule my own flight, I would have booked the flight to Wrocław, but I didn’t have the opportunity to do that. So I flew here\u0026hellip; Oh no, sorry, I flew to Warsaw and I got there at\u0026hellip; The flight was scheduled at 10:00. I could have caught a plane on the same airline, I could have booked a flight here and I would have been here at midnight, but my second flight was also late so I got to Warsaw too late to catch the second flight. I got there five minutes after the flight actually left. So then I’m sitting there at the Warsaw Airport going “how am I going to get to Wrocław?”. So I went down to the car rental places – no cars. Enterprise had no cars, Hertz had no cars, Budget had no cars. I tried my butt off to get here. My only option was to book a hotel. I stayed at the Renaissance right at the airport, and I took a train here in the morning. And I thought “well, it took us three hours to drive here the last time, so it should be a three hour train ride”\u0026hellip; Five and a half hours. So I ended up getting here at 3 in the afternoon, so with all of that all those travel hassles there was no way I could give my workshop anymore. And as soon I missed the first flight, when I landed in Heathrow, I told Damian [Widera] I said: “I just don’t think it’s going to work because I’m gonna have a real hard time even getting there on time, never mind being prepared to speak for eight hours”.\nMS: If you have so many different accidents together, so many delays, maybe you should have played the lottery.\nAB: Yes, probably. In reality, I should have come a day early. For most of my travels when I come over to Europe, I usually book a couple of extra days, so that I can relax and decompress and maybe see parts of Poland that I haven’t seen before. And what I should be doing is doing that at the beginning instead of the end. Then I don’t risk coming in late, right? The other problem is, if I get here early, I want to keep preparing, right? So if I come a day early, I can’t really use that day to go sightseeing, because I’m gonna want to prepare. It’s competing priorities and I chose poorly, so\u0026hellip; But American Airlines did not help me and British Airways did not help me, and LOT airline did not help me.\nMS: You have mentioned that you need to prepare for a session like one day before. But what does the whole process of preparation for a session look like? Do you have some kind of pattern that you always follow? Like starting with a sheet of paper and then writing the subject of the session or there is some other process?\nAB: It depends on the topic but for these topics that involve T-SQL, I build the structure of the demos I’m going to use to show my points in a query file. And then I extract the pieces, the points that I want to get across onto slides. So I don’t think a lot of people do it this way. I’m sure people do but I haven’t heard a lot of people explain it this way, but I work backwards from the demos to the content and that allows me to focus on “what am I trying to show with this demo?”. It doesn’t always work out like I forgot to show a part of a demo today in my session. I mean, I’m human so that kind of thing happens.\nKN: That’s an interesting approach. I’ve never tried that. But that makes sense, sounds very reasonable.\nMS: One more question about preparing for a session. How do you gauge the audience, so what level the audience will be? You have the possibility of preparing the right abstract, but how do you prepare, in case you have purely demos, for tricky questions from the audience?\nAB: I don’t know if I really prepare for that, because things can happen in a session that are just way beyond what you can expect. Erland [Sommarskog] was in my session today and he like attacked me with a question about cursors, because I was talking about cursors and the options in use, and he was: “what does that option really mean?”. I’m like: “uhh, okay\u0026hellip;”. And then I just answered it. I mean, I don’t know if I made him happy with my answer but you just kind of have to be ready to address the question. I’ve had cases where people will just go on and on, and on, and on, and at some point, you just have to shut them down and say: “look, let’s talk about this after the session because nobody else here is getting anything out of your badgering”. I can help you get to this answer after instead of making it a part of the show.\nKN: Otherwise you will finish your session explaining\u0026hellip;\nAB: And that’s happened! I’ve been speaking for 12 years and in my first year, I would speak at the Boston user group. And there was a guy there, I probably shouldn’t say anything about this, but there was a guy there that was very belligerent and very\u0026hellip; like, he demanded answers and he just kept talking and talking and talking. He would like to take over sessions. And it took me a few times of dealing with that guy before I was able to just shut him down and say: “we will talk after, this isn’t the time, let’s talk about this after”. Because he would go off on, you know: “Oracle does it this way”. My talk isn’t about Oracle, so I don’t care if Oracle does it that way.\nKN: So you’ve been an MVP for a very long time.\nAB: Since 1997.\nKN: 21 years, wow. Could you tell us about your journey? I can imagine that the MVP, the whole journey, the situation, and the nomination was probably totally different in those days.\nAB: In the early days, the nomination was: someone who was an MVP said: “This guy should be an MVP too”. And that’s basically how I started. There’s a guy in the Dominican Republic I think, Juan Libre, and he recommended me to be an MVP. I was doing a lot of work on a mailing list and a couple of newsgroups that handled Active Server Pages, if you remember, the original ASP was like a classic. It was based on VB6 and you could do it in JScript too but nobody did. And I helped a lot of people learn how to use that, how to build ASP pages that worked well and performed, and he recommended me based on that and I did that for a couple of years. Then, when asp.net came out, I saw like I couldn’t understand it. I still don’t understand it today. If you asked me to build you a website today, I would use classic ASP, because I know it, and I think I can build it very quickly, even though it’s missing some of the features that have been developed since then. But I saw the writing on the wall. It said: “I’m not going to be able to be an ASP MVP, because that’s gonna go away, and I’m not going to be able to be a .NET MVP, because there’s no way I can get up to speed on this to be in a position where I can help other people with it”. So I started promoting my activities with SQL Server more, because I was doing more work with it, even though I was helping people with ASP and not with SQL Server, I was doing a lot more with SQL Server, so then I just started helping people on the SQL Server newsgroups and helping people solve query problems, and then eventually speaking on SQL Server. And so I transitioned and I asked to have my [MVP] award transition from ASP to SQL Server. I don’t recall what year specifically that happened but it was more than 10 years ago. It might have been 15 years ago now.\nKN: How many MVPs were there in the US at that time?\nAB: At that time? A dozen? In 1997? Maybe more than a dozen. There were probably a dozen ASP MVPs.\nKN: But specifically about SQL Server.\nAB: When I first became a SQL Server MVP, there might have been about 30 in the US. There are a lot more now, I think there are 400 or something, 300.\nMS: At the beginning, it was a small family.\nAB: It was a small family and the other thing that started happening too was the nomination process used to be\u0026hellip; Originally it was an MVP would tell the MVP lead: “you should have this guy as an MVP” or “this girl as an MVP” and now, when I joined the SQL Server group, the SQL Server group was a really tight-knit group and the lead would come to us and say: “this person was nominated, what do you guys think?”. And so we would go around in a circle and have input and say: “well, I saw him have this really bad\u0026hellip;”. It was like Survivor! Like, he would literally say, he would take a vote and say: “how many people think this person should be an MVP?”. Now it has changed quite a bit. Now it’s very quantitative, so the number of people reached, number of sessions given, the number of audiences, how many views your blog gets, how many true retweets your SQL help post can get. They take all these numbers, they put them in a spreadsheet and some person who has no idea of what technology we’re working with or how we’re actually helping people just adds up the numbers, and if you’re above some threshold that they’ve arbitrarily defined, you get renewed. And if you’re not, you don’t. And if there’s\u0026hellip; I have to think, I don’t know how the whole process works, right? They only reveal certain things, but I know that they’ve outsourced the quantitative analysis and I think that they can make exceptions and they can say: “well, this person was below this line but I also know that they’re doing this and this, and this”, which the quantitative analysis isn’t taking into account.\nMS: Has it changed for better or for worse, in your opinion?\nAB: I think it’s changed for worse, only because different blogs that affect different people\u0026hellip; If I have ten people that work at Fortune 500 companies that read my blog post and they go and do something and maybe they buy eight hundred additional licenses of SQL Server because of something I wrote, even though that only counts as one-page view, that led to a huge increase in revenue for Microsoft. Someone else could write something that 6000 people read but nobody did anything, because that wasn’t a good post or it wasn’t something that they took value from. Different audience. So I feel like, by reducing what we’re doing to numbers, takes the quality completely out of it. And whether those numbers are actually valuable or not? I would mention names but there are some bloggers out there that do it just for the impression count, right? They have people that come visit their blog and they’re not getting anything but they go. They have repeat visitors that go and check out the site every day and maybe one in 30 days they get something of value out of it. And if I post once a month but I have half of that traffic come and actually get value out of it, I think that’s more valuable. I’m not saying all my blog posts are better than all of anybody else’s, but I do put a lot of time into the things that I post and I don’t post every day for a reason. If I posted every day, it would be crap.\nKN: Some people just publish every day.\nAB: They’re just trying to trigger a counter or your reader, RSS feed or whatever to go visit the site and get them advertising impressions, and that’s not what I’m in this for at all.\nMS: As mentioned earlier, you’re a speaker. How much do you travel?\nAB: Less than I used to. Before I had kids, I would travel\u0026hellip; one year I did 90,000 miles. I was going all over the place, I spoke at Bits every year for five or six years. Out of Seattle, I did a keynote in Berchtesgaden one year, which is like the weirdest out-of-the-way place, but they had a conference there, I went there. I’ve spoken on cruise ships, I’ve gone all over the place. But then, when I had kids, I went to my company and I said: “look, I need to spend less time on the road because I’m missing out on a lot here”. So this is my only international trip planned this year I think, to speak.\nKN: But the year has just begun.\nAB: Yeah, so in June I’m speaking in Chattanooga and in Orlando. And in the fall, I’ll be in Las Vegas and I’ll be in Seattle. My year is pretty much all scheduled already for travel.\nMS: Do you like travelling?\nAB: I like getting there. I like arriving. I don’t like the in-between part, especially this week. Man, that was hard\u0026hellip;\nKN: So when you travel a lot, what is your approach to the very popular topic of work-life balance? How can you do that?\nAB: I keep in touch, we FaceTime every day, I make sure that, even when I’m travelling, I try to like touch base with my kids every day, touch base with my wife every day. When I’m at home, I try to make sure that I schedule any work that I have to do when I’m not in the office\u0026hellip; I don’t go to the office every day, I work from home multiple days a week. But I try to make sure I schedule\u0026hellip; If I’m gonna work on a presentation that I’m doing in the coming weekend and I’m busy at work during the day, I try to schedule that work for after my kids go to bed. My kids go to bed at 8 o’clock at night, and then my wife usually falls asleep at 9 o’clock, so I’ll do an hour of work from 10:00 to 11:00, usually watching hockey too at the same time. So I try to make sure that I’m focused on them. After school, I pick my kid up from the bus and when everybody gets home, we have dinner and it’s all like “no computers, no phone”.\nKN: Totally focused on your family.\nAB: That’s right. And then we usually watch a show, and we do rock-paper-scissors every night for who gets to bathe the kids and who gets to do the dishes, so we always\u0026hellip; I win most of the time. So there’s always the opportunity where I’m gonna be spending 4 hours a day with my kids, which is important. And I don’t ever want to have to lose that. Travelling less, it’s better. It’s still kind of hard but\u0026hellip;\nKN: Especially if you don’t have international travels during the year.\nAB: Yeah, staying in the country is a lot better. If something were to go wrong right now, it would take me a day and a half to get home, right? Because I’d have to go and try to find a flight and like get home, and it’d be a pain in the butt. If I’m in the US, I can usually get home within a few hours from just about anywhere.\nKN: How old are your children?\nAB: My children are 4 and 6.\nKN: I think you put the link to your purely personal website on your Twitter account?\nAB: Yes, the last time I came to Poland, I went to Auschwitz and Birkenau and Krakow.\nKN: Last year or last time?\nAB: Last time, 2016, and I had posted a whole bunch of pictures there. That was an interesting experience. I wanted to do it, I know it’s not happy\u0026hellip; you wouldn’t take your family there, anything like that, but I think it was important. And the big thing that I got out of that is, I grew up in Canada and they kind of glossed over what happened over here and because Canada didn’t really, not until the end, they didn’t want to have the involvement. And in the history books, you’ll see 6 million Jews were murdered or whatever. And it’s so much more than that. There was so much else that happened other than just “six million people died”. But they really gloss over that and it’s like one sentence in the history books. It’s crazy and I don’t know why that is. And then from the time that I learned about World War II from history class and the time I was an adult, there’s a lot of people saying “oh, it’s a hoax, the Holocaust didn’t happen” and all of this stuff. I just want to tell all these people: “look, if you go there, there is no way you can doubt that that is real”. And if all you have is a history book and your dad’s opinion or whatever, it’s hard to convince people that you need to go there. So I felt like I needed to go and see it for myself. I wasn’t one that doubted that it happened but it’s just so much more powerful having been there.\nKN: Honestly, I haven’t been there but I’m planning to go there as well to see it. Recently, I read in a book about what happened in there, a very interesting book.\nAB: We went to Birkenau first and then we went to Auschwitz. If I were to do it over, I would have done it the opposite way. So if you’re going to go, I would go to Auschwitz first and then Birkenau. I know Birkenau is Auschwitz II but, you know, I would do it in that order.\nMS: What is your favourite part of the Data Platform landscape in your opinion?\nAB: My favourite part of the Data Platform landscape? I like that it’s always changing. I feel that for a long-time SQL Server was SQL Server and that’s all there was, and nothing really changed for like ten years nothing changed. We got a couple of row number functions in 2005 and we’ve got new data types in 2008 but it was still just SQL Server. No big changes. And now it’s moving so quickly, it’s hard to keep up. You kind of have to pick what you’re going to do. I think 10 years ago, you had a lot of people that were SQL Server experts that were experts in SQL Server across the board. You could be an expert in the whole area. That’s not possible now. So you have a lot more specialization and I think out of that you get a lot more innovation in the way that people use SQL Server as a solution. Because 10 years ago it was just: you deploy SQL Server and you run your queries against it and that’s it. And now you have all these opportunities to do things like Power BI.\nKN: PolyBase, Big Data Clusters\u0026hellip;\nAB: Yeah, and all the Azure features, all the Azure services that are coming out. For something we’re working on, we’re using Azure Blob Storage, and if we had to manage our own file system to handle that, there’s no way it would have worked. We would have had to build our own file servers that would have to be huge and have to be highly available and that’s a huge investment. Now that these things are all coming as interconnected services and you can just fire something up and use it, and if you use a lot of it, you pay more and if you don’t, you don’t have to incur that big cost. And I was initially like “all the cloud, yeah, whatever, that’s a fad”. And I think a lot of people were. And I think Microsoft has proven that this is not going away, this is the real deal.\nKN: I’ve seen that many people changed their opinion between now and 4 years ago. 4 years ago they thought: “OK, Azure, it’s not great, we’ll not be migrating in that direction, it’s not possible to migrate a company because of many factors”. But now a lot of people change their opinions. They started adapting to the new situation, I think.\nAB: I think there are some industries where it’ll be really hard. Some people just won’t, they’ll just resist. Banking and anybody with huge data privacy concerns is going to shy away from it, and some people just won’t be able to because of legal reasons. They won’t be able to store their data in someone else’s property. But for a lot of people, I think the cost implications of running your own hardware and running your own operating system, and running your own services, it just goes up on this scale, and at some point it just makes so much more sense to use someone else’s hardware, someone else’s infrastructure.\nKN: Talking about the features, I would like to ask another question: what do you think about SQL Server on Linux, Docker and all the languages that came to SQL Server, like R, Python?\nAB: R, Python and Java I can spell, I can spell AI and I could spell Java but that stuff doesn’t really excite me. But being able to run on any platform, that’s pretty exciting. And being able to run on ARM64, that’s exciting too. For me, specifically being able to run a Docker container that’s running SQL Server on my Mac means that I get half my battery life back. I don’t have to run a Windows VM now. My demos yesterday and today – all Docker. No Windows SQL Server, no Windows VM. Switching between is much smoother. Just docker and one command line to pause, one command line to destroy the whole thing, one command line to fire up a brand new one on a different version, one command line to upgrade. For someone like me who’s carrying this thing around, this laptop around and giving demos and stuff, it frees up resources on my machine where I’m not sitting there waiting, because the VM is taking half my memory. I’m not sitting there waiting for my cursor to respond because all my memory is being sucked up somewhere else.\nKN: So you have a smaller one. Is it 14-inch?\nAB: 15-inch.\nKN: So you should be able to have more than 16 GB of RAM.\nAB: No, it’s capped out at 16 GB, this one. This is a couple of years old. The one after this allowed for 32 GB, but this one does not.\nKN: OK, so it’s hard to run.\nAB: It is! So I was running an 8 GB VM and if you want to do anything fun in there, you can’t. You’re paging to disk and demos take forever, relatively forever.\nMS: What is the personal achievement that you are the proudest of?\nAB: Personal achievement I’m most proud of? Do I have to pick just one? Because there are a couple. One is I was voted Author of the Year for MSSQLTips two years ago. That was pretty cool. And the other is I am the only person on DBA Stack Exchange with the Legendary badge, which is I got 200 – the cap – reputation 150 times, so it’s not really anything except a measure of how many people appreciated my answers on that site consistently over time. Those are two pretty big achievements for me.\nKN: That also explains a little why you have over 8000 followers on Twitter.\nAB: Oh, that I don’t know. I tweet a lot about music, I tweet a lot about hockey, I argue with people about politics\u0026hellip;\nKN: Ah, maybe politics is the reason!\nAB: It could be!\nKN: What hints would you give to young people who wanted to start working in the IT market specifically? What is important?\nAB: Definitely understanding the different layers, the different tiers we have between\u0026hellip; People will think: “oh well, I can just develop an application, I don’t need to care about the database, and I don’t need to care about the network layer, and I don’t care about the data layer”. Those concepts still exist, whether you’re using an on-prem or a cloud service, there are still layers that you have to go through to have apps communicate with data and communicate with the user. And it’s important to not lose sight of that. People will fire up these frameworks and they claim to do everything. They can but with limits, right? A lot of these frameworks are built with the lowest common denominator in mind and they don’t cater to anything else, so if you take any ORM, any mapper, you have this concept where it supports 80% of the things that people would want to do, and if you want to get outside into that other 20%, you’re kind of out of luck. You’ve got to extend the thing or you’ve got to bolt onto it or you’ve got to switch to something else to do something that it wasn’t predetermined people would want to do. Even though they’re very logical things for a certain segment of people to do. They code these statements that cater to updating in a certain way, and if you want to do something where you also want to join to this other thing, there’s no way to do that because they didn’t predict that anyone what would want to do that. And then you end up: “oh well, in this one case, this is this exception rule again, right? In this one case now I have to write a stored procedure and I have to deal with it in a completely different way because this thing has locked me into a certain portion of functionality that is actually possible”. So it’s not all there and I find that when you start packaging this stuff up and extracting away the details behind the scenes of what’s actually going on, and you sugarcoat things with “we’re gonna handle this one thing and this one thing, and you’ll just have to press a button to do that”. And that’s nice if that’s all you ever want to do, but as soon as you want to get out of that, you have to understand all of the architecture or you’re just going to put your hands in the air and say: “well it doesn’t do that”. We’ve been working with Material UI, just for web interface stuff, and I’ll say “well, this one button, it needs to behave this way, when you mouse over it, it needs to behave this way” and the response I get is: “well, Material UI doesn’t do that” and they’re literally putting their hands in the air in a meeting, saying: “well, it doesn’t do that”. But I know that a web browser can do that, so find out how to get around the limitation in Material UI, and then it’s a philosophical argument that you’re not supposed to do things that Material UI didn’t predict you want to do. And I think that’s kind of crap.\nKN: And what about people who just started their journey in IT? Who would like to start? What is the best first step? How to first job for example?\nAB: Whoo, it’s been a long time since I thought about that! Sorry\u0026hellip;\nKN: But I bet you still remember.\nAB: I kind of remember but I’ve fallen into my jobs, like I’ve had two jobs in my career and the first one I was plucked out of school – no interview, no resume, none of that, just plucked and they gave me a job and I worked for them for 13 years. And then I came to SentryOne. That was also no interview. I did consulting for them first and they tried to hire me for about three years before they finally admitted that maybe I didn’t have to move to Charlotte at the time to actually be a productive employee for them. And neither of those involved a job interview. And my initial job, it wasn’t something I went looking for, so I didn’t have this planned thing to go into IT. I taught myself how to build web pages because I could get a part-time job at the school, building the library website for the university, and that’s kind of where I started. So there wasn’t really a formal plan. It was just: “this is easy, I could do this really well” and I started getting on these mailing lists and newsgroups because initially, I couldn’t figure out how to do one thing or some other thing and so I went on there for help and then I realized that most of the questions people were asking, I already knew how to do that, so then I started answering. So it wasn’t really a plan, it wasn’t a set thing of “what should I do?”.\nKN: So you’ve had only two jobs in your life?\nAB: Yes.\nKN: OK, so let me ask a different question. What is it like to be in one job for so many years?\nAB: It’s stable. Like at my last job, I had job security like you wouldn’t believe. I just had this confidence that I was an important person here who is vital to success. And sure enough, when I left that company, it wasn’t because I left, but I left and went to SentryOne and that company just fell off a cliff and disappeared within a year. And again, I’m not saying that was because of me but while I was there, I was keeping all of the things that depended on the database, which was what drove our business. The data drove our business and I was keeping the lights on, and it was just me, I was the only DBA. I tried for three years to hire an assistant DBA and I couldn’t find someone that could do it, so you get the sense of\u0026hellip; and maybe sometimes it’s false, but you get the sense of importance and job security, and that you’re stable here, and it would take a lot to get you out if someone wanted you out for whatever. Being happy at my job, it also relieves any pressure that I have of like thinking about what else is out there. I don’t think about that. I’m not trying to find my next opportunity, because I’m happy where I am.\nKN: I guess you didn’t start from the position of Product Manager at the beginning?\nAB: Correct. My original title was Senior Consultant, and I worked with customers from time to time, but that wasn’t really my main job. I did all kinds of work, but I managed our blogs – I still do. I manage our team blog and sqlperformance.com. So I blogged a lot on behalf of the company and then managed the other authors that we had blogging for us. And I spoke a lot and promoted the company, so I was kind of an evangelist. I was a speaker, I was a community advocate, I did a large number of things and I still do most of those things. They’re just not my primary role. I took over Plan Explorer initially because as a community advocate, it was a community tool, something we were giving out to the community, and it needed a little more hands-on help and getting it out there to people. So I took that over and then I also took over a couple of the other products and a couple of other teams that were working on other products. I’ve been trying to help manage those as well as I can. We’ve got some additional Product Managers who are going to relieve me of some of that work pretty soon, so I’ll have a lot more time to focus on very specific things, like Plan Explorer and our monitoring of Azure SQL Database.\nKN: By the way, thank you for giving us Plan Explorer for free! That was amazing news when I saw it. It was like a year or two ago?\nAB: A couple of years ago it was free.\nKN: But only part of the tool was free.\nAB: Yeah, it was free for a while, it was free for three or four years and then we did some work, we spent a year and a half doing a bunch of R\u0026amp;D and development on some new functionality that makes it much more powerful. Index analysis and history and comments, we added a bunch of things that make it a much better tool to use for query tuning in a team and actually make improvements, not just see what’s wrong but actually help you determine how to fix it. So we introduced a Pro version and started charging for those features. And we did that for a while, and we would constantly have meetings about it and talk about “how is this going?” and “is this really what we want to do?” and we ultimately determined that we shouldn’t have done that and we should have just made all of this functionality free and continue to offer that as if it’s just a free tool. So we took the Pro and the price tag away and just made it all free.\nMS: You have mentioned Azure is on your road map. So what do you think about the trend of migrating to cloud? Is it going to replace all SQL Servers that are on-premise at the moment?\nAB: Definitely not. There are definitely use cases that make sense, where people can move their databases into the cloud but there are more complex\u0026hellip; So if you just have a single database it’s no problem. If you have a couple of databases that interact with each other, it’s no problem. But there are more complex scenarios where that’s not necessarily going to make sense. And as I was saying before, I think that there are definitely some legal constraints around some companies and industry complaints and other constraints, and compliance, various reasons why that might not be an option at all. I think you’re gonna see that grow, I think you’re gonna see that business grow but I don’t think it’s going to eat in very much to the on-prem business. So I think people are still gonna want to run SQL Server in-house. And you’re also going to pick up a lot of customers that have stayed away from SQL Server because it was on Windows. You have houses that are Linux-based and now that they’re going to have a SQL Server option, it opens up more business for Microsoft.\nMS: How will the role of “old school” DBAs change with this cloud invasion?\nAB: It doesn’t change much. I see a lot of doomsday things out there. People are saying: “the cloud’s gonna steal my job”. I don’t think that really changes much. I think the individual, the granular details of what you do is different because if you have Azure SQL Database, you’re no longer dealing with backups. There are certain things that you just no longer have to deal with. You’re no longer dealing with locking down, xp_cmdshell and all these things that are vectors for attacks and misuse. And those things just don’t exist anymore. But you still have to worry about availability. You still have to worry about performance. You still have to worry about all these other things. And I feel like the traditional DBA role, where everything was on-prem and you had to deal with everything, was overwhelming. And I think that a lot of the things that you need to take care of, different companies and different DBAs would just choose “well, I can focus on these three things, these other things will have to wait”, because there’s only so much time in the day, right? And I think that with the cloud taking away some of the routine things that weren’t hard but took up time, I think that’s a smart way to go, because who wants to spend a lot of time doing things that aren’t hard. It’s repeatable and it’s automatable and people don’t take the time to automate it on-prem. Well, the cloud has taken that work away from you and just said: “this stuff is going to be automated, now you only have to deal with these things”. But now you could focus all your time on those things so you can make those things better. So if you have a DBA that’s also the database developer that’s writing queries, now they can focus more on making those queries perform well instead of “we got the results we needed, what’s next?”. Now they can actually spend an extra hour and actually tune that query and build the right indexes and everything else. There’s still so much that\u0026hellip; It doesn’t matter if you’re on-prem or in the cloud, those things still apply.\nMS: What have you learned recently and what is your next item on your learning list?\nAB: Oh, boy. OK\u0026hellip;\nKN: For sure not Java.\nAB: Definitely not Java. Sorry, Java developers, but\u0026hellip; So, I learned a lot more about recovery recently, because SQL Server 2019 has accelerated database recovery, which is the killer feature I think in SQL Server 2019. That’s going to be the feature that makes everybody upgrade. And I didn’t understand the recovery process before well enough to understand how this would benefit. So I spent some time and actually sat down and relearned. Because I learned about it a long time ago but forgot half of it. So I spend a lot of time with that, being more familiar with recovery. What’s next on my list? Kubernetes. I need to learn about these clusters. I got some advanced knowledge of it because the MVP program, they gave us a preview and access to it, but I didn’t really take advantage of that. I think this is going to be, depending on the licensing, I have no idea what the licensing is gonna be like and how expensive it’s going to be, but if this thing is priced reasonably, I think it’s going to be a very powerful solution for customers. Especially things like, think about a flower shop, right? A flower shop that does online sales, FTD, whatever. What happens on Mother’s Day? They go crazy, right? So they have hardware that can handle their worst-case scenario but they’re paying for that hardware all year. And when you have something like this that can dynamically adapt to your volume and how much throughput you’re having, and you’re not paying for that the rest of the time, I think that’s a huge win for any business. So I think that’s going to be something that DBAs are going to have to become comfortable with and companies like us who monitor the things that DBAs care about, we’re also going to have to care about it, and we’re gonna have to learn how to monitor that effectively.\nKN: It’s great that we’re talking about monitoring. So SentryOne is offering the tool for monitoring servers? And Redgate does the same. So do you think these two companies are competitors or\u0026hellip;?\nAB: Yes, I do. And especially\u0026hellip; so we’ve acquired Pragmatic Works recently, we acquired the software of Pragmatic Works and they have a bunch of database developer tools that are similar to what Redgate offers. For a long time, we had competing products, not directly but close. They have a smaller feature set and a lower price point, and ours has more functionality and a higher price point. And we definitely had overlap in our markets but it wasn’t like an eclipse. It was like there’s a section of the market that will only pay up to this much for any kind of monitoring at all, and those were just not customers to us. It’s like Adobe Photoshop. You have Adobe Photoshop and you have a segment of the market out there that’s going to pirate Adobe Photoshop and is never going to be a customer. So it doesn’t matter if Adobe charges a thousand dollars or a hundred dollars for Photoshop, there are people who will refuse to pay a hundred dollars. So it’s the same thing with our business. For the full functionality and the amount of R\u0026amp;D and development that it takes to get our full functionality, we’re not going to let people have it for five dollars. It’s not in our business model. But now, we are trying to go after that lower-end market. Because we realized that companies might have different environments that they’re monitoring, so they have a production environment where these things are really important and then they have a QA environment where it’s kind of important. You don’t need 24/7 life-or-death monitoring but it’d be useful to be able to compare metrics between those two systems. Should we be charging the same price for the thing that you’re doing less with and that you don’t need all the functionality for? So we introduced a SKU – an edition called Essentials, which is 60 or 70% of the functionality but at less than 50% of the price. So you still get a good section of the monitoring, not everything and not all of the R\u0026amp;D we do but quite a bit of it, at a much lower price point, much more competitive with Redgate.\nKN: It’s still cool seeing that the two companies are competitors but you and Grant, for example, are friends.\nAB: Yeah, we’re pretty good friends. I believe he’s gotten in trouble for being too friendly about us. Like someone will ask: “how do I analyze plans?”, “well, we’re the answer”, right? And the same thing for a long time, I would probably get in trouble for this now, but for a long time if someone needed to compare schemas as I would absolutely 100% “go, get Redgate’s Schema Compare and Data Compare”. I think that we have a pretty close equivalent to that, they’re obviously best of breed but I think we have a pretty good solution for that too. But I’m not going to go and take down all of the places where I’ve said “go, get Redgate’s tool” because I still think it’s an excellent tool.\nMS: Also, I think it’s quite healthy for both companies that they are competitors because you look at each other, it accelerates the evolution of the product.\nAB: And it’s better for you, right? Because then you get a better product, whether you pick us or them, you’re getting a better product, because we’re competing. It’s not like the oil, what do you guys pay for utilities? If an oil company treats you badly, what do you do? That’s who gets you your oil, right? This is who provides your electricity. You don’t have a choice. And when you have a choice, I think it drives innovation a lot more and drives people to build better software.\nKN: Anyway, the market for SQL Server is quite small. Small in terms that we know each other. We go to many conferences, you know the people. So from that perspective, it’s small. So, about the blog posts, I’m very curious how long it takes you to write one blog post.\nAB: Oh, it really depends. It’s like how long does it take to prepare a talk. I mean it really depends on how much detail goes into it. I would say a good five to six hours goes into a medium-sized blog post.\nMS: We need to learn from you.\nAB: I mean, I’ve been doing this for a long time, so I definitely have\u0026hellip; not shortcuts, but I cut to the chase kind of in a lot of things. I don’t spend a lot of time worrying about my words. Just like with my presentations. My presentations start from a set of demos and then I build bullet points out of that. Same thing with blog posts. I look at “what am I trying to prove here, what’s the point of my post?” and if it’s to show what’s the most efficient way to trim time from a date-time, I will go and I’ll build the demo environment, I’ll build a table, and then I’ll run the queries that will tell me the answer. And then, once I’ve done that and I’ve determined the answer, now I can go write about it. What I think a lot of people do is they write about it and they sometimes – and I’ve done this in the past, too – you’ll write about something and you’ll expect a certain result and then you’ll go build the thing and you’ll test it and it’ll totally disprove what you wrote. So you spend a lot of time massaging the words and saying all these things, and then you have to go back and redo it because you were actually wrong. Or at least in that instance, that scenario that you created you were wrong.\nKN: So also, from that point of view, it’s better to start doing the demo first. Another reason. OK, Aaron, at the end of our conversation, where can we track you down? Twitter, blog?\nAB: Twitter, I’m @AaronBertrand. So yeah, I’m @AaronBertrand on Twitter and I blog at sqlperformance.com and the blogs.sentryone.com.\nKN: And I think you’re also on Database Administrators StackExchange.\nAB: Yes, Stack Exchange.\nKN: Cool, OK, thank you very much.\nAB: Alright, thanks, guys!\nUseful links Aaron\u0026rsquo;s profiles: Twitter | LinkedIn | MVP\nAaron writes on blogs: SQLPerformance | SQLSentry Blog | StackExchange | SQLBlog\nBooks: SQL Server MVP Deep Dives Volume 1 | Volume 2\nRelated events: SQLDay\n","date":"2019-10-25T07:08:57Z","image":"/uploads/2019/10/ASF026-SQLDay-with-Aarob-Bertrand-1200.jpg","permalink":"/2019/10/asf-026-aaron-bertrand-interview/","title":"ASF 026: Aaron Bertrand interview"},{"content":"Hello folks! 2 weeks passed but eventually, I have some news for you.\nPress Support for Azure Databricks Instance pool for operationalizing Databricks workloads in Data Factory\nPublic Preview feature which helps to reduce cluster start by maintaining a set of idle, ready-to-use instances.\nUsing hash values in SSIS to determine when to insert or update rows\nMerge only actually changed rows when loading to Data Warehouse.\nTabular Editor 2.8.6\nIf you\u0026rsquo;re doing a lot of scripting, this update is definitely for you.\nMicrosoft.Azure.Management.DataFactory Namespace\nThe .Net SDK for Azure Data Factory Mapping Data Flows is out!\nSpendOps with Azure Cosmos DB\nVery good post about Cosmos DB and taming of costs of the service. Look at the video below.\nAutomatically rotate through Power BI report pages on your browser\nVery cool advice if you want to share reports in your company on monitors.\nAzure Data Factory Mapping Data Flows are now generally available\nYeah! Finally!\ntempdb Enhancements in SQL Server 2019\nErin is testing CTP 3.2.\nA word from Microsoft’s CEO Here's my annual letter to shareholders, a look at our tremendous progress and impact over the past year and our collective opportunity ahead. https://t.co/5odAWd3VrY\n— Satya Nadella (@satyanadella) October 17, 2019\rVideo: Real-life SpendOps with Azure Cosmos DB Smile Corner ","date":"2019-10-20T19:53:23Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/10/last-week-reading-2019-10-20/","title":"Last Week Reading (2019-10-20)"},{"content":"T-SQL Tuesday series (the monthly blog party) has been started by Adam Machanic and co-ordinated by Steve Jones. This month\u0026rsquo;s topic is hosted by my friend Alex Yates. He is keen to know what kind of things change your mind. Something that happened in your IT career, but doesn\u0026rsquo;t have to be technically tied. Whatever it was and left a permanent mark on your head. Or you purely just remember that.\nAt the beginning, I started to wonder about what topic I changed or change my mind. Although this doesn\u0026rsquo;t happen very often, I don\u0026rsquo;t think I have a problem with it. To be wrong is human nature. To stick to one post all your life - it may seem like a bad idea.\nAfter all, how do they say it?\nA wise man changes his mind, a fool never will.\nOur lives and experiences are also changing. Constantly. Especially if you work with Azure, or services in the cloud, am I correct?\nThe Person Life is a series of compromises. Should I write a blog post or go for a run? But play with my children first, they go to bed in an hour, after all. Damn, I still have to reply to a few emails and prepare a new podcast episode. There is also a session to prepare for the next SQL Saturday. And also this unfinished book that I was supposed to read a week ago. Why am I writing about all this? Because these things are choices. Priorities. And if you have a family, e.g. a wife/husband and children (optionally) then\u0026hellip; yes. The partner will help us to change our mind. Believe me.\nThe people As I said - our lives are changing. We change these changes ourselves, but also the people we surround ourselves with. Consciously or not - we derive certain patterns from others. I like to be surrounded by many people, I like to observe people, talk about our behaviours and habits because it teaches me something new. It\u0026rsquo;s worth using the patterns you identify with. In this case, it\u0026rsquo;s not about what you thought before. I don\u0026rsquo;t think about it. I think whether the new approach is able to change me for the better, and thus also my surroundings. It\u0026rsquo;s not easy and the process is long. After all, I think it\u0026rsquo;s worth it.\nThe technology From a technology point of view - I don\u0026rsquo;t change my mind very often. Generally, I don\u0026rsquo;t. I also don\u0026rsquo;t remember any specific situation in which I had to do it. Which doesn\u0026rsquo;t change the fact that the architecture of the solution I propose can change over time. The technology around us is changing extremely fast. Take just a few examples from the shore: Azure Data Factory V1 - currently deprecated. What can customers do who have already managed to implement this service? Well \u0026hellip; migrate to V2? Is this a simple process? Definitely not. In fact, I wouldn\u0026rsquo;t even call this process \u0026ldquo;migration|. It\u0026rsquo;s simply to create this process again - regardless of whether we think of ADF v2, Azure Databricks or something else. There is no migrating application or service which does it for us.\nAnother example: let\u0026rsquo;s look at SQL Server. It develops at a dizzying pace. Relatively recently, Microsoft has stated that SQL Server loves Linux. Great. Now, with version 2019 we have Big Data Clusters offering a full cross-section of technologies, including R, Python, Java. Not to mention containers.\nWhether you like it or not - this is the direction in which subsequent versions are heading. Either you accept it and start learning or \u0026hellip; you will stand still and be left behind.\nDo we change our mind in such a situation? I think not - we are simply adapting to the changing environment.\nWhat is your goal? Define it! The problem remains different. With so many services cooperating with each other - it\u0026rsquo;s hard to keep up with them all.\nAnd here we come to the point - where I admit that I changed my mind. When at the SQLBits 2018 conference I realized that I was interested in too many technologies. I realized that I could not be good at everything. It\u0026rsquo;s almost impossible. If you think we can do (MVP people) - then I have to put you right: we don\u0026rsquo;t.\nI changed my mind. I decided to focus only on selected Data Platform services, such as Azure SQL DB, Azure SQL DW, Azure Data Factory, Azure Databricks. These, in turn, entail other languages ​​that should be known: Scala, Python. When I add to this my interest in the SDLC process in the form of Azure DevOps and very helpful PowerShell - it is starting to be too much again.\nNevertheless, I have deliberately decided to give up my interest in other services, although they are really interesting. Don\u0026rsquo;t get me wrong - it\u0026rsquo;s good to have an idea of ​​the different services to know when to use them and what they are suitable for.\nThanks for reading. I hope my post gave you food for thought.\n","date":"2019-10-08T20:30:29Z","image":"/uploads/2017/09/tt_300_11FF5A1D.jpg","permalink":"/2019/10/t-sql-tuesday-119-what-changes-my-mind/","title":"T-SQL Tuesday #119 - What changes my mind"},{"content":"Hello Azure/Power/BI/Data fella! Another two weeks passed very quickly and the new edition of Community conference in the UK is about to begin. Check whether you can attend one of the 5 cities where DataRelay come and take a look at these posts in the meantime. I will be there and talk about ADF Mapping Data Flow for 4 days.\nPress BISM Normalizer: Version 5.0 Released!\n1500 compatibility level, calc groups, M2M relationships, PBI objects, composite models and more!\nPower BI And Case Sensitivity\nIf you may be here about case sensitivity or insensitivity in a dataset, DAX and you want to learn more about them in this post you can read how to work with them.\nADF Adds Hierarchical \u0026amp; JSON Data Transformations to Mapping Data Flows\nWith this new feature, you can ingest, transform, generate schemas, build hierarchies, and sink complex data types using JSON in data flows.\nCommon Azure Cosmos DB use cases\nCheck benefits of using Cosmos DB for retail (web and mobile) applications or as a data store for IoT systems.\nNew Free Azure and SQL Server 2019 Training from Microsoft\nAre you interested in learning Azure or SQL Server Big Data Clusters?\nTOP 5 Benefits of Azure Dedicated Hosts (Preview) for SQL Server Workloads on Azure VMs\nMeet a new hosting model where a physical server is dedicated to your organization and capacity.\nHot patching SQL Server Engine in Azure SQL Database\nMicrosoft Team reveals how do they patch SQL Server Engine without impacting workload.\nSetting a Custom Variable in an Azure DevOps Pipeline with PowerShell\nKendra Little explains how to set and reuse an environment variable.\nA word from Microsoft’s CEO Together with @Novartis, we will apply the power of Azure and Microsoft AI to some of the biggest challenges facing the life sciences industry today. https://t.co/AKjlDFNAHa\n— Satya Nadella (@satyanadella) October 1, 2019 Video: Using Large Amounts of Data in PowerApps https://www.youtube.com/watch?v=0PLznP64O9o\nSmile Corner ","date":"2019-10-06T22:50:59Z","image":"/uploads/2019/09/news-alex-Yvl081TVnvA-unsplash-1200.jpg","permalink":"/2019/10/last-week-reading-2019-10-06/","title":"Last Week Reading (2019-10-06)"},{"content":"It is the first post on series related to installation and configuration of containers on the Microsoft Windows platform.\nThis series is for professionals who start their journey with the containers and use them with Microsoft SQL Server.\nIntroduction As an IT specialist, you are probably familiar with virtual machines that are available on the market for some time. On the market usually, there are two main products that offer virtualization: VMware and Hyper-V. Even though I am sceptical regarding the installation of SQL Server on the virtual machines, this time you can hardly find any SQL Server installed on bare metal – almost all new installations are on the virtual machines. Virtual machines provide many benefits like:\nRunning software on a very specific version of Operating System Experimentation with other operating systems Testing specific version of the application without affecting your Operating System Easier migration between physical boxes Costs saving by enabling the possibility of building highly complex environments on the single physical host etc. Of course, all of the solutions also have some disadvantages – in the case of virtual machines, the main one is the need for running a whole Operating System.\nContainers Imagine that you can run your application without the need of running a separate Operating System – this approach is called containerization. The picture below shows the main differences between virtual machines (sometimes called hardware virtualization) and containers (called operating system virtualization).\nSource: https://stackoverflow.com/questions/16047306/how-is-docker-different-from-a-virtual-machine\nA container is a standard unit of software that contains code and all dependencies. Thanks to that you can easily and quickly deploy from one environment to another.\nDocker As mentioned above, containerization allows to run application isolated from the rest of the system and gives the impression that every application is running on its very own OS instance. Docker is software that enables running containers on a single operating system. In Windows Operating System Docker is based on Hyper-V (thus this role needs to enabled) and shows in Hyper-V manager as Virtual Machine.\nKubernetes If you install Docker on several machines (even with different operating systems) you can leverage the main advantage of Kubernetes – nodes. Each installation of docker starts to play the role of the nodes in the Kubernetes cluster. Kubernetes allows you to automate all of the features of containers such as provisioning, networking, load-balancing, security, scaling, etc.\nWhy use Docker? Let me show a few of the advantages of containerization that can convince you to use containers in professional work.\nImagine that you want to use your application on a different version of the operating system such as Windows, Linux, macOS. Having in mind all of the environmental variables that can cause different behaviour of an application can be overwhelming - just to name different .NET, Java version, even different paths. Using Docker you can package your code to Docker image that is sealed, independent of the Operating System and guarantee the same behaviour on production.\nConcluding, Docker image contains all dependencies, such as libraries, version od programming language, etc.\nWhy use Kubernetes? Kubernetes gives additional benefits related to containers by automation. First, you need to create a cluster of nodes, and then Kubernetes takes control over the cluster and can decide if new containers are needed or some of them are redundant.\nContainers and SQL Server As you have probably heard already, SQL Server starting with version 2017 supports not only Windows Operating System but also Linux. Adding this information with Docker gives new possibilities with an approach to software development and testing. Before it was required to install SQL Server instance on Windows Operating System (described in a blog series from zero to dba), but can be problematic when you need clean installation for testing. On Windows, there is a limit of a maximum of 50 instances on a single OS.\nWith Docker, you can spin up SQL Server using specific version (like RTM, CU1, CU2 etc.) within seconds instead of minutes using the previous approach. This can be extended to certifying if an application is compliant with the new version of SQL Server or more streamlined migration from one version to the another.\nSummary Containers start to be very popular recently due to their main feature – images that abstracts Operating System version, dependencies on programing language version and other variables that can lead to issues on deployment in production systems. With Docker, you can deploy images locally, and with Kubernetes, you can automate the deployment process on the cluster.\nLinks https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/hyper-v-architecture https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/ESXi_architecture.pdf https://stackoverflow.com/questions/16047306/how-is-docker-different-from-a-virtual-machine\nhttps://www.docker.com/resources/what-container# https://kubernetes.io/\n","date":"2019-10-03T10:17:11Z","image":"/uploads/2019/09/belgium-1601918_1280.jpg","permalink":"/2019/10/what-is-containerization/","title":"What is containerization?"},{"content":" Introduction Rob Farley is a Microsoft Certified Master, Microsoft Certified Trainer and is a recipient of the Microsoft MVP Award for SQL Server since 2006. Rob provides consulting and training courses around the world in SQL and BI topics. His community involvement keeps him quite busy - as well as running the Adelaide SQL Server User Group, he is a former Director of PASS and helps co-ordinate many events around the world.\nHe has also assisted Microsoft Learning in developing many certification exams, writes a blog and co-writes \u0026ldquo;Professional SQL Server 2012 Internals and Troubleshooting\u0026rdquo;.\nOn his blog, you can find him writing about various SQL topics, particularly T-SQL and the Query Optimizer.\nThis talk has taken place during SQLDay Summit 2019 in Wroclaw, Poland on 13 May 2019 (Monday).\nInterviewers: Kamil Nowinski (T), Michal Sadowski (T).\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript KN: Hello, Rob! Could you introduce yourself at the beginning of our conversation?\nRF: Sure. My name’s Rob Farley, I live in Australia, in Adelaide. I’ve come to Wrocław to be a speaker at the SQLDay event this year, which is great to be a part of for the first time, to be able to hang out with you all.\nKN: We are after the first day of the SQLDay conference 2019, after your workshop today. How did you find it?\nRF: It was good. I’m tired now because, you know, it’s what happens when you do a workshop all day. I think it’s got an intensity that teaching a three-day class or a five-day class doesn’t really have, because of the pace of it all, the level of content that you’re getting through, the engagement with the audience that you’re getting through, the fact that it’s speaking all day, there’s no lab time and so on, so I find that you’ve got the length that makes it tiring because you’re on your feet all day, but you don’t have time to say “OK, now do this lab”, as you do at a regular training class. So by the end of the day of the workshop, I’m just feeling old and jetlagged. But no, it was a fun day. I do it all without a computer. I’m riding on flip charts, I’m getting people in the audience to stand up, hold them and things like that. I’m getting people up to be able to demonstrate how to run queries as executions plan operators, so it’s a fun day but it’s tiring. I think people enjoyed it.\nKN: Yes, you mentioned two things that I would like to ask about. So the first one is your jetlag. You’re living in Australia now, yeah?\nRF: Yeah, so I had a 27-hour trip I think to get here. I had about 14 hours to fly from Adelaide to Doha in Qatar, and then about 6.5 hours from Doha to Frankfurt, and I had a few hours in the airport there, and then the short 1-hour flight from Frankfurt to Wrocław. I hope I’m saying it correctly: Wrocław?\nKN: Yeah, quite good.\nRF: It looks like raw claw but it’s not, it’s Wroclaw.\nKN: Yeah, perfect.\nRF: For anybody who’s not Polish and listens to this, it’s just like, yeah\u0026hellip; Wrocław makes no sense. But I think I spent like three days learning how to say it. And I’m almost there now, which is excellent.\nKN: Yeah, you already learned it.\nRF: But yeah, it’s a long trip and it’s only 7.5 hours’ time difference at this time of the year because we’ve got winter in Australia and summer here. It’s about the same temperature. Could be a bit colder here at the moment. But the time difference is 7.5 hours but it’s that long flight that just takes it out of you. So it’s only 9:30 here at the moment but it feels like it’s about 11:30 to me. But I’ll probably still be awake at 4 am.\nKN: How did you survive that flight? I still can’t imagine. My longest flight was to the USA and it was like 9 or 10 hours. It was like just enough, you know.\nRF: I left Adelaide in the evening. My flight left there at 10:15 pm it felt like it was an all-night, you know, that 14 hours, and I got to Doha in the morning. The way that I was flying made that it was really kind of all night, I mean getting to Doha at like 6 o’clock in the morning, something like that, because of the time difference and so on. But my problem is that with my height and everything like that I really don’t sleep particularly well. So I had worked all day on a Friday night. I sort of tried to sleep but didn’t. Ended up watching films and reading books and thinking through other ideas, trying not to rewrite my workshop, that kind of thing is always tempting. You’re just going “I should rework that. No! I should leave it exactly how it was”. It was fine, I don’t need to rework it but I think essentially when I got here, I was like “yeah, I’ve been up all night, it’s now about time I arrived in Wrocław”, it was like 6 pm on Saturday and I felt like I’d been up for 40 hours or something like that. It was about 40 hours since I’d woken up on Friday morning until I was landing it was about 40 hours that I was up, and when I landed, I was already ready to sleep. But then I ended up only sleeping about six hours before I was like “Oh, I’m awake!” It’s a cruel, cruel thing, the concept of jetlag and time zones, and so on.\nKN: You don’t know it until you try on your own.\nRF: Actually, it’s way easier to do it on my own than with the family. I remember taking long-haul flights with the kids when they were little, and of course, then you’ve got kids that are awake. And they don’t know that they need to actually still be sleeping because it’s night time outside and so on. It’s like “I’m awake, it’s time to play, isn’t it?” I remember flying with when my oldest child was three, he’s now 21, and becoming a pilot. Today, because I was here, I missed the ceremony where he got his wings. It was heartbreaking to miss it, but this trip has been arranged for over a year and we only found out the date of his ceremony a few weeks ago. But I remember when he was about three and we did long-haul flights between the UK and Australia and having days of him always being awake at night and “I want to play! I’m awake!” I was like: “No, you’ve got to try and adjust”. He’s like “No, I need to be up and playing cause I’m awake”. That’s way worse. It’s way better when you’ve got kids that are old enough to understand the concept of “now you need to be trying to be asleep, now you need to be trying to be awake” instead of when they’re little enough, they just walk along and if they’re tired, they sleep. Travelling by yourself is way easier.\nKN: True. Before we go to the next question because you mentioned that you were flying between Australia and the UK. We met for the first time I think during the MVP Summit.\nRF: I think so, only a couple months ago.\nKN: Yeah, so you mentioned that you were basically born in the UK, quite close to the place where I live.\nRF: I wasn’t born close to where you live. I spent like, the time that I remember living as a child in the UK. I left the UK when I was 14, nearly 15. I did 4 years of high school, all through primary school and so on. But you live in Stevenage and I live close enough to Stevenage to have a Stevenage postcode. It was SG4 for me and you’re SG1 or something like that.\nKN: It’s only 8 miles or something, maybe even less.\nRF: I think about 4 miles from where you are. We used to do our shopping in Stevenage and it was my local cinema, it was in Stevenage. So I remember seeing movies in the 80s in Stevenage.\nKN: It was probably a different city at that time.\nRF: I have no idea because I haven’t been back there for quite a while. I think the last time I was there was\u0026hellip; I don’t know, 2002 maybe? I mean I have been back since, it hadn’t changed in those 13-odd years, in that 17-odd years since I was there.\nKN: If you are in the UK, visit us!\nRF: I still have family in Hertfordshire. I have family down at Bushey, so damn near Watford. So I still see that part of the world when I’m over there. But I can’t say I tend to explore up to the area that I used to live in around Stevenage very much. Maybe one year I’ll explore and see how much things have changed.\nKN: Yeah, absolutely. You’re not curious?\nRF: I’m a little bit curious, it’s just that it’s a long way to go.\nKN: But now you are here, you are very close.\nRF: Yeah, but now on Thursday I’m flying home again because I miss my family and if I’m on something like this I don’t really feel like I get the chance to have an extended time away and so on. So about a year ago, I had a trip to the UK. I went to SQLBits and I saw family, and my grandmother died, and I stayed for the funeral. I was gonna be heading over to the MVP Summit that year, but I went to the funeral. That was a year and a bit ago and I haven’t been back to the UK since. I need to probably arrange more trips to go there. But it’s still a big effort. And I need to take the family at some point, so it becomes an even bigger effort.\nMS: And you have mentioned that your roots are from the UK, but how did you start working with SQL Server? Was in the UK or was it when you moved?\nRF: No, I was still in high school when I moved to Australia first time when I made to Melbourne. I moved with my family because I was still a teenager and I finished high school in Melbourne, I went to university in Melbourne, and I got my first proper IT job. I should say a proper IT job, cause I was working in the computer science department, while I was at uni. So my first IT job was while I was a student working on stuff within the University. But my first post-university job was in a consulting firm in Melbourne. And back in the days of VB 3 and 4 and things like that. And the database was just a necessary part of the application development that we would do for our clients and the consulting and so on. And I started to realize that the database was very important and so on. But I kept being primarily an application development person. I became a project manager and all that kind of stuff. Moved back to the UK for a couple of years in 2001-2002, that kind of timeframe, and then we moved to Adelaide. I feel that way my wife grew up. And still, I was in application development, so the database is part of that but it was mostly on the application development side. And then I kind of got into the SQL community and I started to realize that actually you could be a SQL Server specialist and you could\u0026hellip; The database has always nicely clicked with me. I’ve done pure Maths at uni and so the database concept just kind of clicked. It felt very natural for me. And at the time, I started projecting at things, I started to run a user group. I started to attend conferences and speak at conferences. And over that time, people started to ask me to help them with the SQL Server stuff, and I became a SQL Server consultant, rather than having a career moving through the application development. At some point you’d become a project manager and then you’d become some other level of manager and some other level of manager. The manager stuff never interested me. I was interested in providing services to clients. So when I got the chance to be able to set up my own consultancy and specializing in SQL Server stuff, that just kind work. But that was after I’d already become a SQL MVP. And that was mostly because I started presenting and running a user group and so on in the SQL Server space and being able to explain database concepts to people because I just understood it slightly differently to the way that most did because of the Maths side of things. Because I studied set theory and number theory and logic and so on, and I saw how things relate to each other, and I’ve always seen databases as slightly different to the way that most of other people see them. It’s how I can get away with doing a workshop for a whole day without turning a computer on once. Just a flip chart and drawing concept and getting people to stand and hold pieces of paper and all of this kind of stuff. Because I learnt about what it’s doing and how it’s doing it and why it’s doing it in various ways, and the relationship between things, and so on. And those concepts along the lines of\u0026hellip; if you can actually persuade the computer to run your query the way that you would do it on paper if you could, the fact that the computer will do that obviously much quicker, actually just works out really well. And the fact that you would be prepared to do that on paper if you had to do it on paper, the computer then doing it many times faster than what you would, thanks to a really well running query, as opposed to doing it in some way that you would never dream of doing it by hand because it would never be able to finish. Well, sure, the computer might do it quicker but that’s a poorly performing query. It might take minutes to run when it should be taking milliseconds if you can persuade it to do it the way it would take minutes by hand. So I guess that’s how I got into SQL Server, it was just because I was an application developer and started to see that actually the only thing that really mattered was the database.\nMS: What was the SQL Server version that you started working with?\nRF: Well, my first\u0026hellip; That would probably be SQL 6.0 because back in the mid to late 90s, when I got a job outside of university working for clients, you had databases and so on, that was what one of my earliest clients was running – SQL 6.0. And then I helped him with an upgrade to SQL 6.5. But I was mostly on the application development side. It was just that, you know\u0026hellip; And in fact, interestingly I had one of those clients, the one that we had upgraded from 6.0 to 6.5, I didn’t realize at the time but I developed a cube kind of concept where, for the sake of their reporting, I created tables of aggregates and so on. It was before the concept of OLAP, which was in SQL 7.0. I’d make tables with aggregates and all that so that I could get reports to run more quickly. And I just had no idea that what I was doing was really similar to what became Analysis Services.\nKN: It was natural for you.\nRF: If I’d known at the time, maybe I could have become an Analysis Services specialist back then, but instead I had done that and I kept working in the relational space and all that kind of stuff, and I moved into project management and all this fancy stuff. And then I felt like it was only years later, getting towards the SQL 2005 release when I got into other things like Analysis Services, which was always a fantastic part of the product. I mean, most of the SQL Server things are so relevant to the overall data solution side, but I had no idea that I had been doing BI so long before it was actually a thing inside\u0026hellip;\nKN: So you were doing the aggregation tables even before you the indexed view happened?\nRF: Yeah! It was painful but I looked at various reports that this particular client had and I saw that they were just taking too long to run and so I worked out: “hey, you know what? If I create a table that has these, I can roll them up really nicely and easily, and all that kind of stuff and I made these tiny tables that had the domain aggregates that I could group in different ways and so on, to make these reports run really quickly. I just spent a few hours, in fact, getting these tables ready with ‘this has to be the sum across this and this has to be the sum across that, and this has to be the sum across something else’”. It felt quite painful to process, but I had no idea that what I was doing was really similar to what you did when you process the cube. So yeah, I had an interesting experience overall in getting to become a SQL Server specialist. But even these days I think it’s hard to be a SQL Server specialist because the product is so wide, then how does somebody become an application development specialist. Somebody becomes a specialist C# programmer for example and C# is huge these days, and it keeps changing really quickly and so on. These days people tend to be a specialist in a smaller and smaller area, where the parts are growing around them. And I think in some ways I’ve been lucky that the core concepts around data, whether that’s SQL DB or SQL DW or SQL Server on-prem or hybrid and so on, most of the concepts about what you do with data and how you change stuff with data, and the significance of data within an organization really isn’t changing all that much.\nKN: Yeah, it’s still the same, the same rules.\nRF: A lot of it is still the same concepts. Sure, different kinds of indexes and you’ve got all kinds of different things, but that foundation is way more stable than many other things. You look at how you create a webpage these days, it’s totally different to what you did 20 years ago. Completely different. You’ve got MVC and all these kinds of things and even that’s our path these days. And when I look at what we did 20 years ago\u0026hellip;\nKN: From the database perspective\u0026hellip;\nRF: Exactly, from the database view\u0026hellip;\nKN: It’s still the same, almost the same.\nRF: It could be almost exactly the same database, probably just different indexing strategies and so on and different compression settings\u0026hellip;\nKN: Maybe fewer types, like some of them, are deprecated right now. But the main concept remains the same.\nRF: And I find that in many ways reassuring because at the end of the day people want their data to be reassuring. They don’t want to feel like their data is the one that’s going through all kinds of changes. They want stability because their data is what defines what they do, what they find important and if they lose their data, they struggle to stay around.\nMS: Did you try to explore the world outside of SQL Server, like other database engines?\nRF: Well, when I was doing application development, we were using Oracle and MySQL and things like that. There was a time when actually I felt stronger in PL/SQL than I did in T-SQL, but in some ways, I always prefer SQL Server because it always felt like for me that to be an Oracle specialist that it was some sort of world of dark magic. The list of things you need to learn or that kind of stuff, the Oracle DBAs were somewhere else. And I always felt like they were the ones who were standing in front of the room full of switches and “I’ll go and tweak that up, go and tweak the other up”, whereas SQL Server will perform just as well but will give you way fewer switches to play with, because we understand that it’s gotta be more user-friendly to be able to get a proper foothold in the database space. I’ve always felt like Microsoft knew more about what they wanted to achieve, whereas Oracle seemed like they wanted to achieve the ability to change absolutely any setting and to make sure that to be an Oracle DBA, to be in analytics, but really in any way you needed to make sure that you had dedicated your whole life to the platform and so on. I’m sure these days SQL Server is the wider platform. It’s just as complicated as Oracle was back then. The sheer number of things that come through it.\nKN: Yeah, more products in there, services.\nRF: Yeah, absolutely! SQL Server is more of a platform.\nKN: R, Python, Java\u0026hellip; SQL Server or Linux, Dockers, you know, \u0026ldquo;who cares\u0026rdquo;!\nRF: So many things, SQL Server suddenly has all these different facets to it. It’s just like “really, OK, so I haven’t played enough with that yet, haven’t played with these other things so much yet”.\nKN: I remember always trying to avoid the Java and I just found out myself that Java is in SQL Server now!\nRF: But at the end of the day, I always felt like the Microsoft database platform had a better future than the Oracle one, simply because it didn’t feel like the room full of switches that I could fiddle with. It felt like they had significant switches, not just switches that seemed like they were pointless. So I always thought that SQL Server was the better option. But also I never felt like I was a proper DBA. I’ve never been one for managed services, I’ve never been the person who’s been on call, just in case the box goes down. I’ll happily help people whose databases go down and I know how to\u0026hellip; One of the things I showed today was fixing corruption and things like that, it’s a pure DBA thing. It’s not development process or something like that, it’s pure DBA and I understand a lot\u0026hellip; I can’t think of many things today that I don’t get in that DBA space. I just don’t want to be the person who’s on call, who has to then fix something. I’ll happily answer my phone if it rings in the middle of the night with somebody saying “Oh my goodness, my database has gone down. Can you help?” The answer is always: “Yes, of course”. But I don’t want to be a DBA because the DBA is the person who just makes sure that the system, that the customers, data stay up. And I’m like: “Yeah, but actually this data should be serving the business, you should be able to use it to improve what you do and improve what you know, and it’ll help you make decisions” and all of that kind of thing. The data should be serving, whereas the DBA serves the data to make sure that it’s always there. Whereas I’m a consultant, I help the data serve the business, which is way more fun, I think.\nKN: You mentioned whiteboards. I’ve seen a sentence on your Twitter account and also you used those whiteboards during today’s workshops. What was the idea? How did you come up with that idea to use the whiteboards? Do you use them on regular sessions?\nRF: Actually, in the three sessions that I’m giving in the rest of the week, I won’t even use a whiteboard. One of them is gonna be a discussion. It is a whiteboard in that one, yes. And the one I’m giving tomorrow is me talking about maybe simple things. It’s more like a career development style talk. So that one is just me talking about seeing yourself in movies.\nKN: But there will be a little bit about databases?\nRF: Maybe\u0026hellip; Not much. I might mention Power BI at one point. But this is a session that I’ve given as a keynote before and it’s all about how some of my favourite movies relate very closely to the career of IT professionals, of SQL people and so on. So it’s a fun session but it’s a bit different probably to all of the other talks that are going to be on this week. But the workshop, doing it completely without a computer and so on is because I actually think that when you’re using a whiteboard or a flip chart, demonstrating things in more hands-on, practical kinds of ways, where I’m getting people to hold up pieces of paper, or explaining “OK, so we’ll access data here and then we don’t have all the information, we’ve gotta do a lookup and so we’re going to take that address of the underlying row and we’re gonna go and find that over here” and so on. And explaining to people what’s going on with the internals and showing them “this is what a page looks like” on paper, I find that it kind of triggers different parts of the brain, such that hopefully they will have learned, the people in that room, of course, will have learnt about these internal SQL concepts in a way that might stick with them, in a way that means that they are actually thinking more about the paper concepts, instead of the computer concepts. Because the way to make SQL run well is to pick up the paper, how you would solve something by hand, rather than how you would have done on the computer. If I’m showing the internals of the page, as soon as I do that on a computer, people just say “Oh, what’s that, what’s that, what’s that?” And they get distracted by the amount of information that is there. When I’m explaining that on paper, I’ll just give some header information, I’ll just draw a few squiggles on the page, and here’s the way the data is. And I go “this is the bit that we want to focus on”. And they don’t get so distracted by “Do you have to start in the middle?” And I don’t actually think about filling up 8k of data. I’m just filling up a few things that I’m drawing on the board. I’m gonna say “this is full now, we’re gonna move to the next page”. You demonstrate how a page splits and all that kind of stuff in ways that hopefully people get better or it lives in a different part of the brain because of not using a computer.\nKN: Yeah, and they can focus on what you are sharing and explaining basically in your own pace.\nRF: And without the strange distraction that the computer is to the people sitting in the room. For some reason, if I’m there dancing about in front of them using pens on a whiteboard or a flip chart, rather than sitting at the desk running a query, demonstrating a query plan and saying what’s going on there, if I’m actually demonstrating it to them and getting people out from the audience and saying “Hey, you’ve got data, go and tell that person, go and get that information from them and put it here” and all of this kind of stuff, then that’s helping them see what’s going on in a very different way to actually running queries on a computer. It’s still very useful, but I figured I’ll be different.\nMS: How are you preparing yourself for a speech?\nRF: For my sessions?\nMS: Yeah for sessions, workshops\u0026hellip;\nRF: It’s tricky. To a large degree, it’s about knowing the technical material really well. If I’m going to be doing a session where I’m talking about SQL internals, I want to make sure that I know everything that I care about, right? I need to be able to introspect some of the questions people are going to be asking. If I need to be able to demonstrate how a forwarding rows looks like, how that works, and so on, I need to make sure that I know how that works, I know how I’m gonna explain that, and so on. Knowing that material really well actually means that it gives me a bit of freedom to be able to then design the journey that I’m taking people on.\nMS: So it’s starting with some kind of subject that you would like to present or it’s starting with a clear sheet of paper and then you’re just starting with the concept and then adding something or it’s different?\nRF: Typically it’s thinking about the kinds of things that I’m going to explain to somebody. Often it’s thinking about the conversations I’ve had with people, the things that I’ve explained to clients, the time that I’ve needed to try and adjust somebody’s thinking about something or whatever the case may be. And I can see the story that I want to tell to the audience. I get the concept that I might to explain to them, the mindsets that I’m going to nudge a bit in one direction or another. And then the overall session starts to form because I’m thinking about the ways to explain the things. And I start to feel what can work as ways to explain things and what might not work with how to explain things. And I have a good idea about what I could talk about in an hour and in two hours and so on.\nMS: You have mentioned that you are avoiding making last-minute changes, just before the session. Is it always like that or it depends?\nRF: I always end up feeling a strange amount of stress with presentations. I always feel like\u0026hellip; Let me be clear, I will have how the talk is gonna go from ages beforehand, but as I get close to actually delivering a talk, I always keep feeling like I’m wanting to tweak some bits and so on. And that’s not actually the best way of doing things. You want to be like “no, no, no, this is prepared and it’s fine and we’ll see how you go”.\nKN: You should make a code-freeze three days before your talk starts.\nRF: Absolutely, it’s just that adrenaline about “I’ve got a talk coming up” that makes you want to tweak the thing and you don’t want to be doing that. It’s not good. I don’t even really try to plan new jokes beforehand. There was a joke that I developed for this week because I’m in Poland, where I say to the people in the room: “Oh, so I’ve been learning some Polish. I’ve learned to say ‘cześć’”, which means ‘hi’ and also ‘goodbye’. You kind of move your hand as if to be able to say ‘hi’ and ‘goodbye’, which helps, so that if I say ‘cześć’ and I move my hand like your listeners can’t tell that I am but you’ll know when you look at me, you get that I’m meaning the right word, so even if I’m saying it wrong, it’s OK.\nKN: But even without\u0026hellip;\nRF: I’ve learned to say “dzięki”, which is “thanks”, things like that, and “dziękuję bardzo”, which is “thank you very much”. Apparently I’m saying this OK. And I had a big effort to learn to say “Wrocław”.\nKN: Also perfect.\nRF: And so this thing that I’ve been working on is to be able to say to people “Hey, so I’ve taken the time, I’ve worked with somebody quite closely on this”, to be able to say to you “Hey, welcome to SQLDay, I hope you’re having a great week”, and so I worked with this person, she’s very impressed with my pronunciation, we seem to get a lot out of speaking. So I hope, apparently, this is quite good. “Just let me try this”. And I say: “Jesteś piękną kobietą, masz piękne oczy”, right?\nKN: Nice! Very nice compliment!\nRF: Which of course means “you’re a very beautiful woman, you’ve got lovely eyes”, right?\nKN: Yeah, they told you the truth!\nRF: Rather than “Welcome to SQLDay, I hope you’re having a nice week”. But that’s my joke for the week. And of course I told my wife that this was what I was going to do and she thought it was funny, and gave me the OK to do it. I learned how to say that off YouTube. I didn’t work with anybody to make that work. But the joke works OK. The people in the audience just go “ha, ha, ha, he thinks he’s saying ‘Welcome to SQLDay, you hope you’re having a nice week’, and actually he’s just saying ‘very, very beautiful woman’”. So there’ll be an odd little joke like that that might occur to me a few days before something. But really I’m just making sure that I know the technical stuff, that I’ve got my narrative of the session. So many sessions lose the narrative, right? Somebody wants to have a session and they want to explain some concept and they just go “Yeah I’ll explain everything that I know about some feature” and they make a slide deck where they just list feature after feature and that’s what they’re going to be talking about. And I go: “Oh my goodness, no! Your narrative, you need to know how your talk flows to tell the story to keep them engaged”.\nKN: Otherwise it’ll be boring.\nRF: Otherwise it’s not connected enough to be able to actually get into their brain properly. It’s not just a list of facts. Or a list of features or explanations about features and so on. It’s got to be a narrative so that people see the progression of the talk as it goes on. So in the days before a session, I’m making sure that I know that narrative, I get how I’m going to segue from one thing into the next, how I’m going to develop certain ideas, to be able to lead the people into the next thing and so on. And I feel like I’m always worried that I’m gonna miss something when I move from something into something else. Not to the point that I want to have a slide deck that I’m leaning on too much, to be able to guide that. I want it to feel natural, I want it to feel like I’m having a conversation, I’m leading a conversation as we go. Because then I’ve got the audience, and they’re not being distracted by slides. And if the audience wants to ask questions and starts taking a different direction, then sure, I can now explain that before I move onto that. And I’m not dictated by the slides that I put together, where somebody asked the question and I was like “Oh, yeah, that’s actually my fourth point on this slide, so can we come back to that in a minute?” or anything like that. I can happily answer the question, explain something and then bring it back and work that into the story that I’m telling. But I need to really know the narrative. Preparing for a talk is an interesting kind of concept. And I do it differently I think to most people.\nKN: How long do you travel during the year or do you like travelling?\nRF: I’ve been through phases where I’ve traveled an awful lot and at the moment I feel like I don’t travel that much. For example last year I went to SQLBits in the UK and I have various trips in the States or around Australia. And this year I went to the MVP Summit in the US, it was my first time to the US in two and a half years. A few years ago, when Microsoft was getting me to travel around a lot to teach PDW (Parallel Data Warehouse), and I toured in Malaysia and India three or four times, and Sri Lanka, and stayed at one of the hotels that got blown up recently. And in Auckland a few times in New Zealand. And around Australia as well. So I thought then that I was travelling a lot, but in the last year or so, really not that much. I mean, travelling internationally out of Australia is a big effort. Travelling within Australia is not too bad. And I’ve got employees in Melbourne so I feel like I’m in Melbourne for a day or so quite frequently. And I’ll be there again in a few weeks’ time. But most of my work is in Adelaide where I live, and I try and give those clients a lot of my time. I don’t want to say that I avoid travels, but I try to be available for the community in Adelaide. Because I don’t know, I try and make sure that when I have people in community asking me for help with their databases and things like that, and my clients, of course, asking me with the way they’re doing things, I don’t want to be “Oh, no, sorry, I’m jet-setting off somewhere else that week”. I’d rather be local and be able to help them with stuff. I love the SQL community around the world. It’s great to come and hang out, but at the same time I also like to be home and to be able to be available for the people and see the community where I live.\nMS: Next question is about work-life balance. Do you feel there is something like that?\nRF: I’m not good at taking holidays. I’m really not. I don’t remember the last time I took my holiday. I mean I have three kids, two oldest boys who are both at university and my oldest, as I said, got his wings today, and then I’ve got a big gap before my daughter, who is 11 and who’s just passed her high school. There’s always times when I work too late in the evenings and so on. But I always try to be home and be available for them. I play music instruments and things like that, play my guitar regularly and so on.\nKN: Yeah, exactly. I saw a video of you playing the guitar with Buck Woody, yeah?\nRF: Oh, yeah, back in 2011. That was a long time ago.\nKN: Are you still playing?\nRF: I play guitar, and I don’t write songs about SQL Server anymore, but that was fun for a little bit. But Buck and I are still good friends. I think in some ways it’s still the thing that I’m most famous for in SQL Server world and it’s nearly eight years ago. Go figure. I need to find something different to be famous for. Maybe it’s the flip chart thing!\nKN: Next question, could you tell us about your MVP journey?\nRF: So I became an MVP in 2006, largely because I had taken over running the SQL Server user group in Adelaide about a year and a bit earlier, and had helped run and make it well attended again and I had turned it from getting small numbers every so often when the group ran to having regular monthly meeting and people wanted to hear the stuff that we were organizing and that I was around, including if I presented, I was asked [??]. We got good numbers up, we’ve generated a lot of interest in SQL Server locally and people at Microsoft noticed that and one day I got an email saying “Congratulations, you’ve been awarded”! That was before the concept of “you got nominated and had to describe all the stuff that you do” and all that kind of stuff. That wasn’t how it was back in those days.\nKN: Oh, really?\nRF: Somebody from within Microsoft would recognize that you were influential and one day you would get the award.\nKN: So previously it worked like only the Microsoft employee nominated you and it was a totally internal process.\nRF: There was none of this thing where you got nominated and then you had to fill out a big questionnaire listing all the stuff that you do and all that kind of thing. It was entirely based on who within Microsoft recognizes actually that you’re influencing the community. Back then I was the fifth SQL MVP in Australia. Now we’ve got about a dozen. I think we’ve had it more over the years in between. But the community was quite different then. MVP Summits were so much smaller than the one that was the other month. But you know, Microsoft paid for your flights to get to events like that and it was a very different kind of experience to being an MVP these days. These days it feels like it’s much more\u0026hellip; they explain the things that you can do to try to become an MVP by vlogging and answering questions on forums and all of these kinds of things. In those days, it didn’t feel like there was a prescription to be able to be awarded. It was simply based on if people within Microsoft recognized that you have influence. And maybe they’ll make it happen for you. So I got an MVP award back then and then I got renewed every year since, and I’m still MVP for the time being.\nKN: When was it again?\nRF: 2006. A long time. It’s just one of those things were for me, in many ways the most significant thing is the fact that I’ve run the user group for all that time and some. And the people within Adelaide and you know around Australia hopefully, when they think of SQL Server hopefully I’m not too far from their thoughts around that. And that’s good. It means that hopefully they’ll reach out to me when they want some help and hopefully, I can help them, and so on. I mean, not just for the sake of being able to earn a living from it and all that kind of stuff, but it’s nice to be somebody who people turn to when they need help with it, and when they feel like their data isn’t serving them in the right kind of way when they’ve got trouble with the data, all that kind of stuff. I like to be able to help people and help them be served by their data better.\nKN: Nice sentence. We need to remember that.\nMS: What is the personal achievement that you are most proud of?\nRF: Probably seeing my kids grow to be normal human beings! I don’t know, I think for any parent, you see your kids grow up and you have no idea if they’re gonna survive in life and so on, and I love the fact that my [??] adult son [??] still live at home but I love the fact that they cherish people like my wife and I. They don’t go to the same church that we go to, but they decided to keep going to church and they seem to be on a good path career-wise, with their studies and the avenues they’re looking into to be able to establish careers, which are their own decisions rather than my getting into the database space. But it gives me hope that clearly my wife and I’ve done something right and gives me hope that my daughter, who’s obviously going through her teenage years, will also turn out normal and so on. Cause I see so many people pulling the hair out over their kids and so on. And I like the fact that my boys have survived those years and have a bright future. And to me that’s way more significant than anything on the technical side. That’s what matters.\nKN: Yeah, true. And it’s not easy to be a parent.\nRF: No, and it’s one of the reasons I’m heading home on Thursday morning, because I always feel like if I’ve\u0026hellip; people said: “oh, you should stay longer and look around Poland some more” and so on, and I think “yeah” but really I want to be able to get home and be around the ones that actually matter. If I’m able to come back to Poland one day and potentially bring family and so on and actually have a holiday here and explore the place. And not feel like I’m off somewhere by myself, neglecting my family. I get that I’m here as a work thing, but that doesn’t mean that I shouldn’t be trying to get home at the end of it. As soon as I say I’m gonna spend a few extra days, be a tourist afterwards, I feel like I’m in the wrong place. While the conference’s on, I’m totally fine being here, as soon as it’s done, I feel like I’m in the wrong place if I’m still here. I need to be going home. And that probably relates to the work-life balance thing as well.\nKN: Absolutely understandable.\nRF: It’s one thing helping the client out with stuff in the evening hours, rather than just finishing at 5 o’clock, if that’s what’s needed. But I’m working for the sake of my family. And I’ll be at some conference, when it’s over, if I take a few extra days, that’s not doing anything for my family. That’s just doing something entirely for me, which feel wrong to me.\nKN: And what kind of hints would you give to young people who wanted to start working on the IT market, let’s say.\nRF: That’s really tough. For me, I went to university and I was good at CompSci. I had an interest in the CompSci of the DAI thing. DAI, machine learning side fascinated me at the time and if I’d stayed on, done a PhD, it would have been in that kind of area. But then, during my honours year, I got offered a job in a consulting firm and I took that. The lure of being able to earn money for my family, it certainly helped. And really, that’s what cemented me in an IT career, because I was working in IT. Before that, who knows, maybe I would have got a job in academia or something like that if I’d stayed on a PhD. Do you then become a lecturer, professor or something like that? Researcher? But these days, people get into IT without even going to university and so on. I don’t really know how that kind of works but I would say, for anybody who is getting into IT is to be able to learn what matters. I feel like I spent the first 5-6 or whatever it was years of my career not focusing on data. I was in application development just because that was what my job was doing. And it was only when I got to about 30 years old that I really kind of noticed that actually the thing that matters is the data. Everything else is just there to make mention of data better. The application is so that people can get data into the database. Or to report on the data, or to be able to manipulate the data in the right kinds of ways. The thing that matters is the data. So for anybody who’s getting into IT, think about what matters to you. Maybe it’s the data, maybe it’s all kinds of things. Maybe you like being a sysadmin and you like to make sure the things are up. But for me, the significant thing was the data and I feel like I probably spent too much time not realizing that the data was a thing. But also I would say to people: look and understand the community and the value of community. I spent a long time without even realizing that there was such a thing. I was living in London when PASS had an event there in 2000. I had no idea.\nKN: Like many young people.\nRF: And I think to myself: “Man, if I had known as a 25 year-old living in London, maybe I would have had a very different path, gone into SQL earlier” and things like that. It just could have been very different. But probably the same occurs these days in any career. I would say to anybody, no matter what they’re doing: find a community of people that you can be part of and that you can potentially be an influencer within and you can actually be a mover and shaker within that industry.\nKN: Talk to young men and do networking.\nRF: Sharpen each other and so on, because that’ll help, help other people and that helps you. You’d be seen as a helpful person. That helps career-wise, and all of these kinds of things that people just don’t necessarily seem to get at all, because most people are just focused on themselves when they’re trying to get their career going. And if they actually understand the community aspect and trying to aim at the things that matter, then they’d probably have an easier time.\nKN: How would you describe your current role? Is it more like a dev role, DBA roll, BI role?\nRF: I would describe myself as a consultant helping people with whatever the thing is that they need. Helping them be more effective with their data, helping people do whatever it is they need to do. Sometimes it’s doing DBA type of stuff, sometimes it’s being able to help them with the strategic use of their data, sometimes it’s about the BI side and being able to produce the right report that they need to be able to actually help them with their data. And the type of work that I am doing is very dependent on where the customers are in their overall data story, with their debt maturity, with equality, whether they’re further down the track sometimes, it’s helping people with AI, being able to help them answer questions. And this is across my whole business, it’s the same kind of thing that my employees will do for our customers. It’s helping them do better with their data. It’s not helping their data stay up. We don’t do managed services. It’s just helping them improve. Sometimes it’s database dev stuff, sure. Sometimes it’s tuning and indexed stuff, sometimes it’s migrations helping them get off what they’re on and potentially into the cloud and things like that. Quite often it’s Power BI, and doing Power BI of the stuff and helping people get into their data and understand their data better, and sometimes it’s AI and we’re helping people ask questions with the data and train models and being able to then say “OK, now I know, what are you going to do with this now that you know this answer? What are you going to do, if we can get an answer for you? What’s happening with all of this?” But it’s still all part of that overall data story.\nKN: It’s generally Data Platform.\nRF: Absolutely, it’s Data Platform, it’s Analytics, it’s all of that. But I don’t call myself a DBA or DB developer or whatever. I just call myself a consultant. Because I’m happy to consult whatever they need.\nMS: What do think about the current trend of migration to the cloud? Is it going to replace SQL Server on-premises?\nRF: I think the location of the infrastructure is becoming increasingly irrelevant. It used to be that if you needed a SQL Server environment, you used to have to just ask the IT department in your organization and they would arrange it for you. And these days it’s really not that different. You’re asking Microsoft or Amazon or whatever to be able to provision your a new SQL Server. You’re just asking a different crowd. Maybe you’re still asking the IT department and they’re provisioning one for you in the cloud instead of on-prem. To people, they shouldn’t care where the data is, how it’s being managed and so on. That should be left to the people who are happy to manage that. And whether they’re doing that using cloud or whether they’re doing that on-prem, it shouldn’t matter to the people that the data is actually trying to serve. It matters for the people serving the data, it doesn’t matter for the people whom the data is serving. And that’s where I prefer to live. With the people that the data is serving, helping them be served better.\nMS: What have you learned recently and what is the next item on your learning list?\nRF: I don’t have many items on my learning list. I tend to find that\u0026hellip; I mean I see new features come through and I go “oh, I want to be able to learn more about that” and so on. And I feel like that’s a continuous thing, but I never write lists of “I need to check this off, I need to check this off, I need to learn about that, I need to learn about the other”. I think there are areas that continue to evolve, that means that the list of things to learn within that space is worth developing some more. And I think the intelligence area is big with that, whether it be R Services or running a Python within the thing. Constant training models and all of that kind of stuff is something that I want to continue to develop. Because I’ve done plenty of that kind of stuff but I feel like it’s still an area that I think you could spend a very long time making sure that you’re an expert in. You could go cave diving in that stuff for months, for years, as it continues to evolve just in front of you. But at the same time, the things that I want to learn is more about my clients. Whenever I’ve got a new client, I’m trying to, you know, “that’s what I’m wanting to learn about, what are they trying to do with stuff, what can we put in that’s going to make that work better?” As I learn about the clients, I end up going “oh, I actually know what Azure Functions might be good for them, because the bit that I’ve heard seems to actually fit”. And then that leads me to dive a bit deeper into the tick to really find out whether it does suit the client and see whether actually I’ve got things that might not work particularly well and are less of a match. But the driving thing is always “how can I get to know my clients better” and “how can I get to know some tech better”. That’s the thing, the thing that makes me interested is what’s happening with the people, rather than what’s happening with the tech.\nKN: That’s better than\u0026hellip; the ideas and the needs about learning comes from that part of business or area.\nRF: People are way more interesting than tech. Tech moves quicker for sure but the people are interesting. Tech is just tech. People are that matter.\nKN: At the end of our conversation, tell us where we can find you.\nRF: Probably in my room, sleeping. But don’t find me there.\nKN: Tell us your room number.\nRF: I’m room 101, which means I’m in\u0026hellip; to quote the book 1984 that’s the room that you get sent into when you need to just be disappeared. I think it’s quite appropriate that I’m in 101. But that may fits. But when I get back home from this trip to Poland, I’ve got a few weeks until SQLSaturday season in Australia. Some of my Red Gate team will come again for some SQL in the City Down Under events across Australia and New Zealand. And I’m definitely going to be in Melbourne for the SQLSaturday in mid-June. I’m tempted but not definite to go to Brisbane as well on the 31st of May, it’s only a couple weeks away.\nKN: We’ll not publish this interview that fast. We know we can find you on many conferences like SQLSaturdays as you mentioned. What about the Internet? Where can people find you on the Internet?\nRF: Oh, I tend to find that if people search my name plus SQL, they tend to be able to find me on Google searches, but I’m on LinkedIn, I’m on Twitter as @Rob_Farley. I’m easily emailed, my company, LobsterPot Solutions, of course, is very findable and I’m always happy to\u0026hellip; anybody can DM (send Direct Message) me on Twitter for example. I don’t have the thing of only people that I follow can DM me. No, everybody can send me messages.\nKN: But you know that not everyone can DM you until you are following them.\nRF: No, I’ve got a setting that says “anybody can DM”.\nKN: Ah, you can set this up? I didn’t know.\nRF: Yeah, that’s a setting you can choose. So I’ve had that for a while now. So anybody can DM me.\nKN: Sorry, I was not aware.\nRF: You need to change that setting, depending on whether you want people to contact you or not.\nKN: I don’t mind as well.\nRF: I understand that a lot of people don’t want to let just anybody send them a message on Twitter. For me, no, I’m fine with it. But people can track me down that way, they can send me emails and so on. Especially if they say “oh, I had an interview with you or I saw you at a conference or something like that”, then I’ll definitely try and answer them.\nUseful links Rob\u0026rsquo;s profiles: Twitter | LinkedIn | MVP\nRob\u0026rsquo;s company: website | Blog\nRelated events: SQLDay\n","date":"2019-09-26T23:02:08Z","image":"/uploads/2019/09/ASF025-SQLDay-with-Rob-Farley-1200.jpg","permalink":"/2019/09/asf-025-rob-farley-interview/","title":"ASF 025: Rob Farley interview"},{"content":"Hello #datafolks! Let\u0026rsquo;s review what happened last week in Data World. It\u0026rsquo;s hard to admit but I missed last week\u0026rsquo;s post due to my visit to Sweden (SQLSaturday) and not too many read post.\nPress One-On-One With Satya Nadella Full Interview\nToday data is everything. Worth a watch!\nThe Unkillable DBCC CHECKDB\nCheck when you can see ROLLBACK of CHECKDB session\u0026rsquo;s operation.\nsp_WhoIsActive Is Now Open Source\nIn fact, it happened a few months ago. Thanks to Adam Machanic for open-sourcing the code.\nSQL Server Change Stream\nDebezium provides a modern, easy to use, Change Data Capture experience.\nMinimal Logging with INSERT…SELECT into Empty Clustered Tables\nPaul White explains what kind of role DMLRequestSort plays in here. Great demo!\nOrdered queue processing in Azure Functions with Sessions\nProcessing at scale might (and for sure do) cause out of order.\nCheck out how Cognitive Services can work along with Azure Search.\nAzure portal September 2019 update\nRegular update from Azure portal.\nMicrosoft Professional program is ending\nSad to see Microsoft Professional Program discontinued.\nGDPR Consent\nMake sure you (and your company) do understand what that does mean.\nCommon Data Warehouse Functionality Redefined By CTAS\nGreat review of all scenarios you should use CTAS in Azure SQL DW.\nA word from Microsoft’s CEO Tools and Weapons from my colleagues @BradSmi and @CarolAnnBrowne is out today. It’s an important book on some of the biggest tech issues of our time and their impact on all of our lives – and well worth reading. https://t.co/N9KL0oycbV\n— Satya Nadella (@satyanadella) September 10, 2019 Video: PowerShell in Azure Data Studio https://www.youtube.com/watch?v=Wan6gC2ui-U\nSmile Corner ","date":"2019-09-22T11:34:36Z","image":"/uploads/2019/09/news-alex-Yvl081TVnvA-unsplash-1200.jpg","permalink":"/2019/09/last-week-reading-2019-09-22/","title":"Last Week Reading (2019-09-22)"},{"content":"It\u0026rsquo;s time to announce and disclose our next guest of the podcast \u0026ldquo;Ask SQL Family\u0026rdquo;.\nLadies and gents, please to meet Rob Farley, who has come to us from Australia. We were talking with him during SQLDay 2019 in Wroclaw, Poland.\nThe 25th episode gonna be rolling out next Friday and in the meantime - find out fragments of the conversation in the following short video: https://youtu.be/KNeXeHInbiY\nThanks!\n","date":"2019-09-21T18:39:29Z","image":"/uploads/2019/09/asf025-video-invitation.png","permalink":"/2019/09/asf-025-video-invitation/","title":"ASF 025 video invitation"},{"content":"Hello all and welcome after my one month of my absence. What was I doing? Thank you for asking! I have been on holiday. ONE month of very relaxing, bit activity (running, walking on mountains) holiday. Oh, I almost forgot I drove over 4,000 miles. But. Already returned from holiday and grey prose of life mode has been switch on. I commenced a new job on Monday, 2nd of September, which is an exciting thing and, simultaneously, a very demanding period for me right now.\nNevertheless, let\u0026rsquo;s back to the weekly series and duties.\nPress SQL Server 2019: Not your Grandpa\u0026rsquo;s SQL Server\nBob Ward (T) summarizes what version 2019 brings.\nAzure Cosmos Capacity Calculator\nEver wanted to know the RU cost of just 1 read and write operation in Azure CosmosDB, without writing code?\nUnderstand star schema and the importance for Power BI\nDimensions, facts, SCD and more according to power BI.\nVisual Studio Code: Develop PySpark jobs for SQL Server 2019 Big Data Clusters\nVS Code is all you need.\nMulti-protocol access on Azure Data Lake Storage (preview)\nBlob APIs now work with accounts that have a hierarchical namespace.\nReturn results to Caller\nCheck out this tweet how to pass something from Databricks to Azure Data Factory as a result of an activity.\nSQL Server 2019 release candidate refresh with Big Data Clusters\nRC refresh for SQL Server 2019 is now available to download.\nWhat’s new for SQL Server 2019 Analysis Services RC1\nVery well summarized up by Christian Wade.\nA word from Microsoft’s CEO We are building Azure as the only cloud with limitless data and analytics capabilities across our customers’ entire data estate, bringing hyperscale capabilities to our relational database services. Take a look. https://t.co/0Pw9fZuRCq\n— Satya Nadella (@satyanadella) September 5, 2019 Video: Docs on Azure SQL Database Machine Learning Services https://www.youtube.com/watch?v=jyD2j3d-lmU\nSmile Corner ","date":"2019-09-08T23:43:53Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/09/last-week-reading-2019-09-08/","title":"Last Week Reading (2019-09-08)"},{"content":"In a week time from now, I will be landing in Gothenburg, Sweden. SQLSaturday #898 will take place over there and I gonna speaking as large as life.\nI\u0026rsquo;m doubly happy because I will be there for the first time - either the city and the country.\nBig thanks to Mikael Wedham who gave me that opportunity!\nI gonna speak about Azure SQL DW during the session titled:\nMove part of your body to Azure SQL Data Warehouse If you never touched Data Warehouse in Azure it\u0026rsquo;s worth to understand MPP (Massively parallel processing) and see why a migration might resolve some issues and when is not a really good case. I explain the differences between traditional, on-premise Data Warehouse and the one in Azure. Obviously, as always, the live-demo is a must-have.\nPlease come and see me presenting this topic if you’re interested in.\nAgenda As always, SQL Saturday events are free, so feel welcomed.\nThe event in Gothenburg has 18 sessions in 3 parallel tracks, classified on the following categories:\nApplication \u0026amp; Database Development BI Platform Architecture, Development \u0026amp; Administration Cloud Application Development \u0026amp; Deployment Enterprise Database Administration \u0026amp; Deployment Click here if you wanna check full agenda of Saturday.\nWhy not join us?\n","date":"2019-09-06T15:54:59Z","image":"/uploads/2019/09/gothenburg-hall-800.jpg","permalink":"/2019/09/speaking-at-sql-saturday-898-gothenburg/","title":"Speaking at SQL Saturday #898 Gothenburg"},{"content":"It\u0026rsquo;s Wednesday, 31st of July 2019. I will remember this day for a few reasons.\nNot only that but the last 18 months I spent working with amazing people, in a unique atmosphere and work culture, where you feel that you can always count on the help of your team members. I\u0026rsquo;m really glad that I have been part of that.\nI will miss out all of them.\nLeaving ASOS ASOS logo Working as a Senior Data Engineer at ASOS, I have learned a lot over the last 1.5 years.\nOn the battlefield, I had to deal with such Microsoft Azure services as Azure SQL DW, Azure Data Factory (including Mapping Data Flow), Azure Databricks and more. I started learning Python and Scala when migrating current solutions to new environments in the Azure cloud and when building other processes, often called with buzzword #BigData.\nThis is where I, as a member of the team, successfully implemented deployment automation using shiny-new Azure DevOps for databases (SSDT), Integration Services (SSIS) packages and helped with much more\u0026hellip;\nI will remember the ASOS company for one more special reason.\nIt was during my work there, at the end of a long day filled with meetings, I sat in a corner of the audience room and started looking at emails on my mobile. Then, I was thrilled when I saw that email:\nGreat news. MVP Award. Soon thereafter, we ended the day with a joint venture to the pub, where I shared this awesome information (for me) with the team.\nThe following photos have been taken on that day.\nThanks to you all, including all analysts I have been working with!\nHoliday When I decided to finish my contract in ASOS, I haven\u0026rsquo;t known where I go afterwards.\nAlthough there were some conversations with few companies - they were ongoing with no rush.\nI knew that I wanted to spend the whole of August on holiday with my family and then begin something new or start looking for a new opportunity. So, I was not really looking for a new job actively. The job found me.\nJoining ALTIUS Altius logo Namely: A very convincing lady, who really wants me to join the company. She was very patient answering my questions during many of our conversations (greetings to Sara!).\nAlthough the decision was not easy (you know what I mean if you ever work as a contractor or self-employed person),\nthe offer given was very tempting. I made it. I\u0026rsquo;ve been hunted. And I feel good. Why?\nI\u0026rsquo;m starting a new job at Altius as a Principal Microsoft Consultant.\nNew people, new challenges and a new position - all these factors will make this an exciting journey. I am convinced that everything will be fine.\nI still will be working purely with Microsoft technologies utilizing and developing my skill sets even more.\nI\u0026rsquo;m also very glad to admit that Altius will be supporting me during conferences, user groups, and other community activities, whereas I will do my best to help them growing and providing even higher quality and better services to their customers.\nI love win-win situations.\nFurthermore, I\u0026rsquo;m very proud that I will join among other friends MVP awarded in there:\nJames Dales (Practice Director) and Paul Andrew (Principal Consultant \u0026amp; Architect) who has recently revealed his transition from Adatis.\nAlso, I can not wait to meet other Altius employees. It is an amazing feeling to be able to work in such a good company.\nTough to say goodbye Something ends, something new begins.\nChanging the workplace is often difficult. For most of us and it\u0026rsquo;s normal, it\u0026rsquo;s OK. We get used to people, especially if they\u0026rsquo;re good at working with them. But time goes on, the environment changes, especially in the cloud.\nWhat I am happy about is that with some, I will still see each other during events. My activity will remain unchanged or even increase. Although all this involvement in the community and work for free may seem strange to some people - that\u0026rsquo;s what makes me where I am and I have so many great friends from #sqlfamily. I recommend everyone to get out of their comfort zone and do \u0026ldquo;something more\u0026rdquo;, something \u0026ldquo;beyond\u0026rdquo;.\nDo not say you do not have time. If you do not have time - do sleep less. Do you need to sleep for 8 hours? All right then, just sleep faster - as Arnold Schwarzenegger, one of my favourite actors, said.\nIn the meantime, I take off on holiday and see you soon!\n","date":"2019-07-31T06:46:33Z","image":"/uploads/2019/07/ian-schneider-TamMbr4okv4-unsplash-1200.jpg","permalink":"/2019/07/the-next-crucial-step-in-my-career/","title":"The next crucial step in my career"},{"content":"I believe that most of you have got a holiday already (enjoy!) or are planning some now. This is the last post of \u0026ldquo;Last Week Reading\u0026rdquo; series before my holiday. I have not decided yet what I will be reading during the holiday: posts, audiobooks, paper books (fiction or technical)? Time will show.\nPress SQL Server 2019 community technology preview 3.2 is now available\nThe latest CTP version of SQL Server 2019 is out.\nContinue a Foreach loop after an error in a SQL Server Integration Services package\nWhat is Propagate Variable in SSIS responsible for?\nUnderstanding Allocation Units – In Row Data, LOB Data \u0026amp; Row Overflow Data\nLet\u0026rsquo;s take a look into the internals of SQL Server storage.\nThe OUTPUT Clause for the MERGE Statements\nFind out what does OUTPUT contains when MERGE is used.\nBudget vs. Actual Model in Power BI: Calculations and DAX\nDo apply DAX to calculate monthly, daily or weekly, even if your budget data is at a quarterly level.\nOptimizing Clustered Columnstore Indexes\nMaintenance of CCI is even more important than common indexes. Check out why.\nSQL Managed Backups and Operating System Error 87\nVery good article about the reasons and action to be taken to avoid this issue.\nTutorial: Monitoring Azure Databricks With Azure Log Analytics And Grafana\nWhat kind of monitoring for Azure Databricks? There is a recipe step by step.\nFiltering the Top 3 products for each category in Power BI\nLearn how to apply a TOP N filter for each Category in Power BI.\nA word from Microsoft’s CEO Wonderful to have so many customers with us at our campuses around the world this week to collaborate and innovate with our employee teams as part of our annual hackathon. https://t.co/OEZuaza6Tt\n— Satya Nadella (@satyanadella) July 25, 2019 Video: What is Azure SQL Database Hyperscale? https://www.youtube.com/watch?v=Z9AFnKI7sfI\nSmile Corner ","date":"2019-07-28T14:28:25Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/07/last-week-reading-2019-07-28/","title":"Last Week Reading (2019-07-28)"},{"content":" Introduction Leila Etaati is an AI MVP, mentor, trainer, speaker and consultant. She spoke in some important conference such as Microsoft USA Ignite, Microsoft Data Insight Summit, SQL PASS, SQL PASS BA, Difinity. She has over 10 years’ experience working with SQL server. She was involved in many large-scale projects for big sized companies as SQL Server and BI consultant. She worked in Industries including banking financial, power and utility, manufacturing. Leila is Ph.D. of Information System department, University of Auckland, New Zealand. She is a lecturer of BI and Machine Learning courses. Leila is an active Technical Microsoft AI blogger for RADACAD.\nThis talk has taken place during MVP Summit 2019 in Redmond, WA (US) on 22 March 2019 (Friday).\nInterviewers: Kamil Nowinski (T), Marek Matuszewski (T).\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript Marek Matuszewski: Hey, everybody. Thank you very much for joining our podcast, channels, Limcore and SQL Player. We are today at MVP Summit in Redmond together with an MVP in AI, in Data Platform category, Leila Etaati from New Zealand. Hi, Leila!\nLeila Etatti: Hi! Really excited to be here. This is actually the MVP Summit. All of the friends in Data Platform and AI are here, so this is really exciting. And also lots of awesome sessions, new things happening in these products. That’s so cool. I’m really happy to be here.\nKamil Nowinski: Including the weather, right?\nLE: Oh my god, the weather is so amazing. No snow, and\u0026hellip;\nMM: Program Managers, they ordered it exactly for us!\nLE: They said that and that’s true.\nMM: Leila, tell me how long it took you to come here to Redmond. Because initially, I thought when flying from Australia, you always have to go East, or West\u0026hellip;? Wait a second, you flew West?\nLE: Yes, that’s correct. Actually, I’m based in Auckland, New Zealand and there is a direct flight from Auckland to San Francisco, that is about 12 hours, and then from there to here, about 2 hours, actually. So together it’s about 15 hours. Not bad.\nKN: Not so bad, I thought it’s even worse.\nMM: I spoke today with guys from Brazil and it took them 25 hours.\nKN: That’s bad.\nMM: Think about the jet lag!\nKN: There’s no jet lag at all.\nMM: But what I saw last time on Facebook, on one of your accounts and Reza Rad’s\u0026hellip; You are from RADACAD, you can tell people what RADACAD is. But tell me, you have a dog.\nLE: Yes, we have a big dog. 47 kg\nMM: Who’s caring about the dog now?\nLE: He is in dog care now, unfortunately. He is a big dog, and he is an American Akita. It’s 47 kg. He’s very fluffy, very playful and we really miss him!\nMM: Akita, wait a second, it’s Japanese?\nLE: Exactly, it’s a Japanese breed. Akita breed can be American Akita or Japoaneese Akita, Khersi (our dog) is American Akita. I can send the picture so you can actually share it with others. Yeah, he’s so cool!\nKN: Tell us how long you’ve been living in New Zealand?\nLE: About 7 years.\nKN: If I remember correctly, last year you both bought a new house in New Zealand?\nLE: Yeah, we have a new house in New Zealand. Actually about 45 minutes away from the city. It’s an area named Whangaparaoa. It’s a really nice area. It’s a bit far from city and the noise.\nKN: Yeah, we’ve seen fantastic pictures of your house on Facebook.\nLE: To be honest, New Zealand is really beautiful, so no matter where you have a house, there’s a lot of nice views there. I really enjoy living in New Zealand! But for sure Whangaparoa is one the nice area in Auckland.\nMM: And tonight, actually today, depending on the time zone, we are in Redmond, in Microsoft’s headquarters, at the MVP Summit. So maybe some people don’t know what the MVP program is. Can you describe in your own words what the MVP program means to you personally?\nLE: Sure, so the MVP program actually is a chance for people who are excited about different Microsoft products. They come and gather, and connect with the product team and the product team provide new features that they are going to deliver in near future. Also, the product team can get feedback about what these products or idea from MVPs. So the MVP program is a connection between customers, experts and the team. We (as Microsoft MVPs) work with customers, we present new features and exciting tools to people in sessions, we writing blog posts or other activity that we do, and then we receive the feedback from end-users, developers and other, so we are a kind of channel that provides these feedbacks to Microsoft people it. Also, it’s really good for us because we get to know what’s important about the product, what’s coming and also it helps us connect with really good friends. They are like family to us. For example, we have the SQL family, Data Platform family, Power BI family, so yeah, that’s a kind of second family.\nMM: So we all know those two things, so firstly the expertise we can get from other people and of course the network. It is extremely interesting meeting people here in Redmond who you usually watch on YouTube or read blogs and then you can just shake their hands and\u0026hellip;\nKN: Approach, say hello, talk, raise some questions, it’s amazing.\nMM: It’s a great opportunity that we can meet all together in one week. I wrote on Twitter a sentence that this week Redmond has the highest density of MVPs in the world.\nLE: All of the MVPs from around the world, from Asia, Europe, South America, everywhere. People from Australia, New Zealand are here, so it’s really a reunion of MVPs. It’s really exciting to hear different languages MVPs from all around the world. People talk in German, French, it’s so exciting. I really love it.\nKN: Leila, please tell us what you do on a daily basis, in terms of business with Reza.\nLE: We run a company named RADACAD. I and Reza are co-founders of it. There are some people helping us to run the company as well. and both of us are Microsoft MVPs. It’s a small business so we do training and consulting on Power BI and Microsoft AI. Reza is more focused on Power BI, I’m mainly focused on AI and Microsoft AI. That doesn’t mean I’m not doing Power BI. I do, but my focus is actually beyond that. Beside that, we run several conferences in Auckland, we are running SQLSaturday, Global AI Bootcamp, Difinity and off course User Group meeting. Also, we write blog posts, every week we try to have a blog posts about new features in Power BI or Microsoft AI and also webinars, so yeah, we are quite busy.\nKN: I’ve seen your blog and I’m still impressed that you’re keeping up the frequency of new posts.\nLE: Thank you so much!\nMM: But apart from all those community activities, blog posts, being at so many conferences, do you offer for your clients consulting services or training, or do you build ready products for them? What is your focus?\nLE: Our focus is mainly on training. We also have consultancy people but that’s actually ad hoc consultancy, it’s not a continuous one from 8 to 5. It’s not like that. And most of them are remote. So we have customer all around the world. Sometimes, we even have customer meetings at the airport. So yeah, we’re used to that lifestyle. Or we write our blog posts sometimes at an airport, on a plane then publish it when we access to internet. However, some time we access to a good internet connection in flight like Air New Zealand flight from Auckland to Seattle provides a great internet connection\u0026hellip;\nMM: Very busy people.\nKN: Apart from that, how often do you attend conferences across the world?\nLE: I can say that we spend about 5 months a year in travel. That’s a lot.\nKN: That’s like 40% of your time.\nMM: And Kamil, have a look, despite such a busy schedule and travelling a lot, you’ve managed to write a book about machine learning with Microsoft technologies. You’re now writing a new book but before that tell us what your last book is about.\nLE: The first book that I wrote is about R in Power BI in 2017. About 200 pages and it was about how you can do R and other things with Power BI. From visualization to machine learning. That’s a free book that’s available for everyone. Another one was again a free book that I published recently. It’s also free. It’s Azure Machine Learning, it’s about 100 pages. I talk about working with Azure Machine Learning Studio. And the third one that’s gonna be published hopefully next month is about machine learning with Microsoft technologies. So it looks at different products, from cloud to on-premises and what features we have, how we can do that. So, for example, it looks at how we can do machine learning in Power BI with R or Python or we can do that in cloud technologies like Data Lakes Analytics and also Databricks or Azure Machine Learning Studio or HDInsight or Spark. So I try to provide a view of different possibilities of machine learning regarding the technologies that we use.\nMM: And what do you think about pre-built models, I mean the cognitive services from Microsoft?\nLE: Oh, thank you for mentioning that! Also, one of the chapters is about cognitive services, how we can use it inside other tools like Power BI and also Bot framework, so I have twenty chapters already and the last chapter looks at when we should use each of them and for what purpose.\nMM: What do you think from a business perspective: which of those cognitive services might be the most useful?\nLE: You mean which services of cognitive services?\nMM: Yes, because we have a lot of them, about 60 services.\nLE: Yeah, I can say all of them are useful in any place, but I want to provide an example. For example, for text analytics, sentiment analytics, we can use it in Power BI to see how much our customers are happy about our products. Or for example another one that is about keyword extraction, we can find out what someone’s main purpose is, for example in email or product reviews. Also for the image detection, if you have data that is an image, you can classify the image in other apps. So all of them are useful in real business.\nMM: Actually, you mentioned a very good topic, sentiment analytics or text analytics are very useful in campaign planning, marketing, brand management and social listening. Those are exactly the topics we support our clients in Cluster Reply, where I work for. And the other topic which I personally find very interesting is age research, for knowledge mining topics. It’s so powerful that you can\u0026hellip; actually for all your documentation you have printed or manuals, extract those images, extract those tables and understand what is in them and make a search on top of that. And to make it even more attractive, there are pre-built models, there are pre-built tools. There are actually puzzles we just have to combine. It’s brilliant.\nLE: Yeah, and it’s good because you have some customers who are business users. They don’t want to write any R or Python code. And we have people who deal with code, they want to do machine learning writing in R and Python. So for both of them we have different tools in Microsoft products, which is really cool. For example, for Power BI recently we have a version of doing AutoML and Azure Machine Learning Studio without writing code in Power BI services. So many people with a business background, without knowing anything about machine learning, they can do it.\nKN: Yes, I’m happy that we are talking about R and Python because we know that recently Python came to Power BI. What is your opinion on R and Python, which language is actually more popular?\nLE: That’s a tricky one actually.R is for statistical analysis, so all of the things that are related to statistical analysis, R is really good. Python is good mainly used for deep learning and image processing. So they have lots of good libraries on that. But there is always a fight between R people and Python, which one is better. But I think both of them can be useful.\nKN: OK, let’s go back to the beginning. Tell us when and how you started working in the market. Which product was it at the beginning of your adventure.\nLE: First, after graduating from Computer Science, I was a C# developer. I also worked with database, so we designed applications, business layer, access layer. And I found that I’m more interested in the database layer rather than interface layer, so at that time I started working as a database developer and also later on I started going into Business Intelligence, so I started with SSIS and SSRS and SSAS. And gradually, I got familiar with Power BI Desktop. This was really awesome. And actually, I started working with Power BI Desktop and that was really good. But in the meantime, I did a PhD in University of Auckland and I used their recommendation system and I was in the process of designing a recommendation system. So what I actually did was I created a tourist recommendation system using collaborative filtering and content base filtering algorithms and that was the point when I said “Oh my God, I think I love machine learning”. So I started self-learning all the time. R was so popular, and I started learning R and Azure Machine Learning Studio myself and gradually I’ve gone to other services. So that’s about 2014 when I started my machine learning / data science journey.\nKN: OK, so from the very beginning it was Microsoft SQL Server, the database you mentioned?\nLE: Yes. C#, SQL Server, BI, SSIS, SSRS, Power BI\u0026hellip; Actually, Power Query, Power Pivots, then Power BI Desktop and also in meanwhile machine learning tools.\nMM: That’s very interesting how you develop yourself, how you build additional skills on top of that, which is always easier if you have one skill and it’s always easier to put another skill on top of that. I want to ask about the new book that is coming. Can you tell us something more about what it’s going to be about?\nLE: It’s going to published with Apress, the one that has 20 chapters about different Microsoft technologies. There is another book that I’m working on and it’s more about deep learning with Microsoft products. I’m still creating the table of contents for that one. That’s at a starting stage.\nMM: And in the context of Microsoft Technologies, they evolve and develop very fast, there are many new features coming, especially in machine learning stack. What do you think is the most difficult part of writing such books about such new technologies?\nLE: The update. So for example, in Power BI there are lots of changes, the same happens in machine learning. New technology comes and they do a great job so new things happen all the time. I should be careful about the changes that happen, so I can reflect them. For example in 2014 we just had Azure Machine Learning Studio. It was easy to catch up with new technologies. Now we have ML .NET, we have DevOps for Machine Learning, we have Azure Machine Learning Services and many other things. So it’s a bit hard to keep up with new things happening. That I think is the biggest challenge.\nMM: Has it ever happened to you that you were almost ready with a chapter and then came back to it and said “OK, I have to correct it because an update came” or something?\nLE: Yes, Databricks actually did with a previous book, so I was introduced to Azure Databricks last February and at that time the content had already been finalized, but I had to include Azure Databricks in my book. Or AzureML Workbench, there were some changes happening to that so I needed to update it.\nKN: OK, so writing a book is one skill. Another one is how you can learn all those things. So reveal this secret to us: how can you do that, how do you find time for that?\nLE: Oh my God, I think you guys are doing the same, so I think you know\u0026hellip;\nKN: But maybe everyone has a different approach to it.\nMM: Has their own secrets.\nLE: For me, I have different things. One is some inspiration to learn. Attending these conferences and seeing what’s new. And also, I tend to speak on specific topics. So I say “I want to speak on this topic so I need to go around and find out everything about what I’m going to talk about”. So that’s another motivation for me to learn. So when teaching others, I need to learn myself. And also, writing blog posts I need to dig deeper to know the whole area. And also these tools are like a game for me. I love to go and play with them, see how they work. Sometimes I cannot say it’s work. It’s more of a hobby for me. So that’s exciting.\nMM: It’s also the topic we talked about before we got on air: know something about everything to have broad knowledge, the whole scope, the big picture but what is more important is to have one topic where you know everything about something. So know something about everything and know everything about something.\nLE: Exactly. So I try to focus more on AI, so I do AI in Azure Machine Learning Services and Studio and others but also I’m trying to have a look on how we can do AI in Power BI, AI in PowerApps and Flow. I also have a look on Power BI in Data Platform also. I try to focus more on AI but I also know what’s happening in Power BI, PowerApps or Flow.\nKN: My next question is: how do you prepare for a speech at a conference?\nLE: The first thing is I provide an agenda, what I’m going to present. If it’s a demo that I’ve already created, I’m going to practice again to see if no changes happened because they may change the product sometimes. So I make sure it’s the same process. And regarding the speaking, it’s become kind of part of my life. There is always some stress. Even if you’ve spoken a thousand times, you have some stress. But I love being on stage, in touch with people. That motivates people to learn and use that product and provides lots of energy for myself. So I really love that.\nMM: You said that you make so many speeches, you have less time because of it but do you still have time for clients to give them services, to provide with training? Are your and Reza Rad’s clients big companies, partially departments or groups of people, or rather individuals worldwide who have similar interests to you but are at a very early stage?\nLE: Both, actually. We have big clients who need some advice on the product development that they have. Or they come to our training on Power BI or Machine Learning. Also, we have some people who are good at those things but they need some more advice on the development. So both, actually.\nMM: Do you already provide training on Databricks or are you not ready yet?\nLE: Actually, I provide it in my trainings. I have a 3-day training about machine learning in Azure products. That includes machine learning services and also Databricks in that. But there are other possibilities that I didn’t cover yet, but machine learning – yes.\nMM: Because as we all know, Databricks is developing so fast, it’s changing. I can tell you, with Cluster Reply with have a project with one client together with Databricks Program Managers who are adjusting Databricks itself just to perform the queries that the client required.\nLE: Databricks is awesome, I like it. It’s so good, there are lots of enhancements happening.\nKN: What do you think about DevOps in your area. Do you think customers are ready for that or DevOps itself is ready in that area, services like machine learning for Databricks, Azure, other services?\nLE: I’m actually not a DevOps expert but regarding machine learning, yes. Because machine learning uses lots of algorithms and lots of libraries and different data sets so it’s definitely one of the needs, if you want to look at machine learning in production stage. That’s a really great process happening in Microsoft Technologies regarding that. Lots of things happening. I cannot talk about them due to my NDA but great things happening.\nMM: Maybe you mean the new TFS service? [laughs]\nKN: Mainly about the process, how to deploy exactly and go live for production.\nMM: It’s a joke.\nKN: What hints would you give to young people who would like to start working in the IT market, whatever area?\nLE: I think it’s a good idea to be in touch with the community. Volunteering different conferences, being in touch with the people who attend the sessions in that area, looking for what excites you. Anything that makes us excited and we see “Oh my God, I love this”, we can be really good at it. So love and excitement about one topic can help us be good at that, earn money and everything.\nMM: Would you then say that you work 24 hours a day or you don’t work at all? You know what I mean?\nLE: To be honest, sometimes I cannot distinguish exactly between work and hobby. I don’t know if what I’m doing is a hobby or work. Mostly when I’m writing blog posts or speaking, I say “I didn’t do anything today!” I did for example a webinar, I wrote a blog post but I feel like I didn’t do anything because I always imagined that work should be kind of very frustrating but actually it’s not.\nKN: So you’re not having to think about your work-life balance? Because it’s mixed in one box, right?\nLE: Yes.\nMM: Actually, my mother told me when I was little: “Marek, find something that’s fun for you, that will be your passion, and you will not work a single day in your life”.\nLE: Yes, she said it well, that’s the perfect one-sentence explanation. You’re right.\nKN: On one hand it’s true but still, it still counts as work, so\u0026hellip;\nLE: Yeah, sometimes we have some problems, for example with customers but still, some of the customers are also really excited about what they’re doing so their excitement transfers to myself. So I feel lots of energy to help them. Also, I have lots of students, I do lots of mentoring. We have lots of people who start a job, a new start-up and they want some advice. So we also allocate some mentoring time as free for them. Or for universities. I’m also in touch with some universities. For example, they used to teach students about SQL or BI stuff. Now, they’re moving towards machine learning. So I also help them through developing, so that’s also really exciting. I didn’t earn money at that time but I really enjoyed it. It’s kind of my hobby.\nMM: I have a question to both of you actually because we wanted to make it a discussion: have you already met in your career or work environment millennials? How do they approach their jobs, tasks? What about their patience in achieving something? You know what I mean?\nKN: Yes, I think it depends on the country. I’m originally from Poland but I’ve lived in the UK since 2015 so I think it’s totally different. What I see in Poland is much more different from the UK. From my current perspective of the client I’m working for, I think young people right now, they kind of have a different approach to private life and work. They really focus on work in terms of business working hours but also they really want to live their private life, so it’s very important for them to distinguish these two things, two areas apart. I think it’s good.\nLE: Yeah, I have the same idea.\nMM: What I was mentioning is, to be honest, from the perspective of time, what we have right now, about the technologies which we are developing, what I meant about millennials is that actually, to be honest, I wish I was a millennial, I wish I was born later. You know why? Because today, it is so easy to get knowledge, which actually is everywhere.\nKN: That’s true, but you were already born much later than me so\u0026hellip; [laughs]\nLE: We can do face recognition to see that!\nKN: I did! And it lied about my age, but I’m happy because it showed that I’m 33 years old, which is not true, but OK, let’s not go into this topic. I wanted to say that when I was learning new things, going to school etc., there was no fast Internet, no Google at that time. I’m not sure if you remember those times\u0026hellip;\nMM: You know what I mean, it was more difficult, knowledge wasn’t so accessible as it is today.\nKN: It was really hard to find any materials if you were looking for something. It was really hard. I still remember that I spent days searching in many different search engines when there was no Google. It was very hard.\nMM: Leila, what does the sentence “Sharing is caring” mean to you?\nLE: You mean about the community? That’s actually really helpful. Let me provide an example. When I was exploring, using cognitive services in Flow, I got stuck in the connection between OCR and PowerApps. Actually, I searched everywhere, I searched YouTube and I found a really great SharePoint person in Christchurch. He does a lot of these things. So because of him I was able to connect that. He shared something publicly with everyone on YouTube and then I could actually use it. And I always recommend it in my blog posts and sessions. And I add something to that, add other things. He shares something with me, I add to it, I share it with others, I write more blog posts. That helps people. I think it’s good.\nMM: What I personally think about this topic is that contributing to the community and helping other people solve issues actually firstly makes me happy but also develops myself.\nKN: Absolutely, yes. That’s why I’m since\u0026hellip; I don’t even remember when I became part of the community, part of a local group in Wrocław, Poland. I started helping as a volunteer at the biggest conference in that part of Europe, which is SQLDay. It was other things, other activities, then I started my own blog. All those activities consume a lot of time but I still happy to do that because I know that people can learn something new. I can learn something new because when we’re talking, that’s fine, but when you record something and publish on the Internet, it will stay there, so you need to be careful what you write and you should double-check what you write about, and then you need to learn. If not from scratch, you need to at least refresh your memory, refresh your knowledge, double-check everything, not to share wrong things.\nMM: Leila, when are you coming to Germany, to my Microsoft AI user group?\nLE: Oh, I’d love to! I’m looking forward to that, so let’s see in the future, in the coming year. I’d definitely love to come, absolutely. Thank you! And also, I’d really like to have you guys in New Zealand, in Auckland and for Difinity.\nKN: I’d like to come sometime. Maybe not this year but maybe next year, who knows.\nLE: Sure! The weather is really nice in February and that’s a perfect time to be in New Zealand.\nMM: What is the weather like in New Zealand in February?\nLE: It’s our summer so the weather is perfect for everything. This year, after the event we went fishing with Charles Sterling and many other friends there. That was so fun actually, you should come!\nKN: Last time, because we met previously in London at a local group. I asked you about the conference and you told me, I don’t remember when it was exactly, but I remember that the call for speakers started very early and closed very early as well so that was a problem for me to raise that topic and plan in advance.\nLE: You’re right. So maybe we need to advertise more on that. And also because we need to provide the session so people can see and register. And the attendees are really happy about the quality of the sessions and they’re already happy about the speakers. They’re looking for all of them and new ones actually.\nKN: You have a lot of MVPs at your conference as far as I remember.\nLE: They help us, they are great. Thanks to them.\nKN: At the end of our great conversation, could you tell us where we can find you and Reza and how we or someone else can contact you or ask about something?\nLE: Our Twitter, that’s me, it’s @leila_etaati. And also through our email: leila@radacad.com and reza@radacad.com and @RezaRad on Twitter. Also on LinkedIn. We try to be responsive.\nMM: Or simply Google: “Leila MVP”.\nLE: Actually, we have another Leila. She’s an Excel MVP, so we have two Leila MVPs.\nKN: Obviously, we’ll put all the links in the post.\nMM: OK, Leila, Kamil, thank you very much for coming to this great conversation. I think it’s time for our next session at Microsoft’s headquarters in Redmond. Let us enjoy the last official day, actually, there are post-days, Thursday and Friday also at the MVP Summit in Redmond.\nKN: Thank you very much.\nLE: Thank you, it was great, thank you!\nUseful links Leila\u0026rsquo;s profiles: Twitter | LinkedIn | MVP\nDifinity Conference: Website | Twitter\nRADACAD: Website | Leila\u0026rsquo;s email\nLeila\u0026rsquo;s books:\nAnalytics with Power BI and R (online, free book!) Machine Learning with Microsoft Technologies Related events: MVP Summit\nBonus: Video! You did not expect that, right? So do I. Marek was recording everything and we have got a video of this meeting too.\nhttps://www.youtube.com/watch?v=l3nNjLcYhl8\n","date":"2019-07-26T20:42:10Z","image":"/uploads/2019/07/ASF024-MVPSummit-with-leila-Etaati-1200.jpg","permalink":"/2019/07/asf-024-leila-etaati-interview/","title":"ASF 024: Leila Etaati interview"},{"content":"Another week passed. Look what did I find during that time.\nPress British Airways faces record £183m fine for data breach\nThe penalty imposed on BA is the first one to be made public since GDPR rules were introduced.\nAzure Architect Expert Study Notes\nA good summary of knowledge if you are preparing for the Azure exam.\nIncrementally shrinking a large SQL Server data file using PowerShell\nTry this script if you REALLY need to shrink big database - chunk by chunk is a better idea.\nAzure Jumpstart\nExactly: how to learn, practice and certify in Azure.\nPowerShell in SQL Notebooks in Azure Data Studio\nRunning PowerShell script hidden by the markdowns sounds brilliant. Rob [T] shows us that.\nPower BI Desktop July 2019 Feature Summary\nWhat\u0026rsquo;s new in PBI in July?\nAzure Data Factory Mapping Data Flow for Datawarehouse ETL\nDesign your own pattern of Slowly Changing Dimension (Type I) in ADF, step by step.\nA word from Microsoft’s CEO Together with @ATT, we will apply the power of Azure and Microsoft 365 to transform the way AT\u0026amp;T’s workforce collaborates and to shape the future of media and communications for people everywhere. https://t.co/vxSBaeH9Hy\n— Satya Nadella (@satyanadella) 17 July 2019 Video: Taiwan\u0026rsquo;s Linker Networks A new way to give driving lessons for autonomous vehicles\nhttps://www.youtube.com/watch?v=k1Jv_N6Aoyg\nSmile Corner ","date":"2019-07-22T07:32:31Z","image":"/uploads/2019/05/news-absolutvision-365898-unsplash-390.jpg","permalink":"/2019/07/last-week-reading-2019-07-21/","title":"Last Week Reading (2019-07-21)"},{"content":"\nI\u0026rsquo;m happy to announce that the next episode of \u0026ldquo;Ask SQL Family\u0026rdquo; podcast is coming.\nThis time our guest is an amazing woman, not only in AI and Power BI area, Leila Etaati. We were talking to her during MVP Summit 2019 on Microsoft\u0026rsquo;s campus.\nAre you interested to get know her point of view of AI \u0026amp; Machine Learning nowadays?\nWhat is the reason for existing both R and Python languages?\nDo you know what kind of secret Leila disclosed to us?\nCheck these things in the next week and in the meantime\u0026hellip;\n\u0026hellip; simply click the PLAY button above or move to the YouTube: https://youtu.be/7DKRJTyaL6E\nEnjoy!\n","date":"2019-07-19T06:00:19Z","image":"/uploads/2019/07/img_5d30ea5d59d8c.png","permalink":"/2019/07/asf-024-video-invitation/","title":"ASF 024 video invitation"},{"content":"Hello, data eaters! Today, coming back from my friend by car, I was browsing a few posts on my cell (calmly - my wife was driving). At some point, I realized that today is 7/07. Yes, a special day for this blog. It\u0026rsquo;s his birthday. The second! As time goes on \u0026hellip; I should prepare some statistics for this occasion \u0026hellip; but I didn\u0026rsquo;t do it. I\u0026rsquo;m glad that I still have the strength and joy of running this blog, so let\u0026rsquo;s wait another year \u0026hellip; will you wait with me?\nPress Azure Databricks. The Blog of 60 questions. Part 1\nTerry (T) and Simon (T) started this series recently.\nIntroducing UTF-8 support for SQL Server\nPedro Lopes, Microsoft (T) about the internal representation and performance differences between UTF-8 and UTF-16.\nAzure Cosmos DB: A competitive advantage for healthcare ISVs\nBuilding a solution for real-time tracking and management of medical devices? Azure Cosmos DB can help here.\nUploading files to Azure Data Lake Storage Gen2 from PowerShell using REST API, OAuth 2.0 bearer token and Access Control List (ACL) privileges\nMichal presents all details from the end-to-end scenario. Many notes, examples\u0026hellip; prepare some time to read.\nChaos engineering for SQL Server running on AKS using KubeInvaders\nKilling pods in a game,\u0026hellip; I meant\u0026hellip; Kubernetes? Yes, let\u0026rsquo;s learn something and play!\nQuick Tip: Implementing “IsNumeric” in Power Query\nFind an expectation of how and where you can use this function in Power BI.\nCreating Basic Greater Than Or Less Than Rules With Power BI Conditional Formatting\nIf you want to know little more how to implement basic \u0026lsquo;greater than\u0026rsquo; or \u0026rsquo;less than\u0026rsquo; rules.\nA word from (former) Microsoft’s CEO Nuclear power is the second-largest source of low-carbon electricity in the world — but its growth is slowing down. This important @IEA report outlines how countries can support existing reactors and advance nuclear innovation. https://t.co/cS9cuHn1WA\n— Bill Gates (@BillGates) 3 July 2019 Video: https://www.youtube.com/watch?v=PZ6cNht_XmI\nSmile Corner ","date":"2019-07-07T23:35:33Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/07/last-week-reading-2019-07-07/","title":"Last Week Reading (2019-07-07)"},{"content":"In this post, I will quickly show you how to create a new Databricks in Azure portal, create our first cluster and how to start work with it. This post is for very beginners. The goal is to build that knowledge and have a starting point for subsequent posts which will describe some specific issues.\nAzure Databricks Azure Databricks is a fully-managed version of the open-source Apache Spark analytics and data processing engine. Azure Databricks is an enterprise-grade and secure cloud-based big data and machine learning platform. Databricks provides a notebook-oriented Apache Spark as-a-service workspace environment, making it easy to manage clusters and explore data interactively.\nAzure Databricks offers optimized spark clusters and collaboration workspace among business analyst, data scientist, and data engineer to code and analyse data faster.\nCreate Databricks in Azure portal Let\u0026rsquo;s start with the Azure portal. Firstly, find \u0026ldquo;Azure Databricks\u0026rdquo; on the menu located on the left-hand side. If you can\u0026rsquo;t see it - go to \u0026ldquo;All services\u0026rdquo; and input \u0026ldquo;Databricks\u0026rdquo; in the searching field.\nAzure Databricks Service in Azure Portal Then, click the \u0026ldquo;Add\u0026rdquo; button, which gives you the opportunity to create a new Databricks service.\nPut a required name for your workspace, select existing Subscription, Resource group and Location:\nNew Azure Databricks Service Select one option from available in Pricing Tier:\nStandard (Apache Spark, Secure with Azure AD) Premium (+ Role-based access control) Trial (Premium - 14 Days Free DBUs) Right above the list - there is a link to full pricing details.\nPremium Tier is giving you more control about who has access to what.\nFor our demo purposes - do select \u0026ldquo;Standard\u0026rdquo; and click \u0026ldquo;Create\u0026rdquo; button on the bottom. Creating a new cluster takes a few minutes and afterwards, you\u0026rsquo;ll see newly-created service on the list:\nList of Azure Databricks instances Simply, click on the service name to get basic information about the Databricks Workspace. Click \u0026ldquo;Launch Workspace\u0026rdquo; and you\u0026rsquo;ll go out of Azure Portal to the new tab in your browser to start working with Databricks.\nAzure Databricks Service Page First glance at Azure Databricks The browser opens a URL address similar to the following:\nhttps://northeurope.azuredatabricks.net/?o=4763555456479339#\nAs you can see - there is a region as a sub-domain and a unique ID of Databricks instance in the URL.\nThe landing page of Azure Databricks is quite informative and useful:\nAzure Databricks Landing Page We\u0026rsquo;re going to focus only at a few sections now, located on the left:\nAzure Databricks - landing page. A good exit point to various features. Workspace - here you will create notebooks in your own or shared folder. Clusters - list of defined clusters. A cluster is a computing unit that can execute our notebooks. Jobs - create and schedule jobs in order to run the selected notebook at a specified time utilise require cluster. In this post, we will focus briefly at Workspace and Clusters.\nWorkspace: New notebook Let\u0026rsquo;s create our first notebook in Azure Databricks. You can do this at least two ways:\nAzure Databricks Common Tasks New Notebook Workspace Shared (folder) Arrow Create Notebook (see image below) Azure Databricks: New Notebook Then, name the new notebook and choose the main language in it:\nAzure Databricks: Create Notebook Available languages are Python, Scala, SQL, R.\nI said \u0026ldquo;main\u0026rdquo; language for the notebook because you can BLEND these languages among them in one notebook. That means you can use a different language for each command. Furthermore, MarkDown (MD) language is also available to make comments, create sections and self like-documentation.\nI named my notebook \u0026ldquo;LoadCSV\u0026rdquo;, so let\u0026rsquo;s put the first line of code in there (cmd 1):\nAzure Databricks: First command In my case default language is Python, but if I would like to apply another language for a specific command, the change must be declared in the first line:\n%scala\r%sql\r%r\r%python For MarkDown language use %md and then write down things in MD in that cell:\nAzure Databricks: MarkDown in command (edit mode) Once you click outside of the cell - the code will be visualized as seen below:\nAzure Databricks: MarkDown in command (view mode) Let\u0026rsquo;s add more code to our notebook. As an example, we will read a CSV file from the provided Website (URL):\nimport pandas as pd\rurl = \u0026#39;https://www.stats.govt.nz/assets/Uploads/Business-price-indexes/Business-price-indexes-March-2019-quarter/Download-data/business-price-indexes-march-2019-quarter-csv.csv\u0026#39;\rdata = pd.read_csv(url) Adding another few steps (commands):\nPressing SHIFT+ENTER executes currently edited cell (command). Try to do this on the first cell (print \u0026lsquo;Hello world\u0026rsquo;). Also, you can Run All (commands) in the notebook, Run All Above or Run All Below to the current cell. All these and other options are available on the right-hand side menu of the cell:\nAzure Databricks: Cell\u0026rsquo;s options But, before we would be able to run any code - we must have got cluster assigned to the notebook. Let\u0026rsquo;s create the first cluster.\nCreate a cluster Select \u0026ldquo;Clusters\u0026rdquo; and click \u0026ldquo;Create Cluster\u0026rdquo; button on the top:\nAzure Databricks: Create a cluster A new page will be opened where you provide entire cluster configuration, including:\nCluster Name Cluster Mode - Standard or High Concurrency Databricks Runtime Version - choose one of predefined option. Python Version Enable autoscaling - I disabled this option for demo purposes. I don\u0026rsquo;t want to spin up too many nodes and pay for them. Terminate after: I set up 15 minutes (of inactivity). After that time pass - cluster will be automatically terminated. Worker Type - size node of each worker and the number of active nodes. I need only one. Driver Type - coordinator node Azure Databricks: New cluster Once you click \u0026ldquo;Create Cluster\u0026rdquo; on the above page - the new cluster will be created and getting run. Since that moment you are charging for 2 nodes (VMs) - driver node + 1 worker. At any time you can terminate the cluster leaving its configuration saved - you\u0026rsquo;re not paying for metadata.\nRun notebook The last thing you need to do to run the notebook is to assign the notebook to an existing cluster. Go to the notebook and on the top menu, check the first option on the left:\nAzure Databricks: Assign cluster to notebook Choose a cluster you need. If the cluster is terminated - you need to run it first\u0026hellip;\nAzure Databricks: Start assigned cluster \u0026hellip; before click Run All button to execute the whole notebook.\nWait and check whether the cluster is ready (Running):\nAzure Databricks: State of cluster Export/Import notebook Each notebook can be exported to 4 various file format. In order to do that, select from top-menu: File -\u0026gt; Export:\nDBC Archive Source file IPython Notebook HTML The code presented in the post is available on my GitHub here. Intentionally, I exported the same notebook to all format stated above.\nYou can utilize Import operation when creating new Notebook to use existing file from your local machine.\nConclusion As you can see - writing and running your first own code in Azure Databricks is not as much tough as you could think. Next challenge would be to learn more Python, R or Scala languages to build robust and effective processes, and analyse the data smartly.\nIn further posts of this series, I will show you other aspects of working with Azure Databricks.\nUseful links Platform Release Notes\nDatabricks documentation: Training \u0026amp; FAQ\nAzure Databricks pricing\nMicrosoft Learn: Azure Databricks\n","date":"2019-07-04T01:27:17Z","image":"/uploads/2019/07/andrew-buchanan-p6ttJiJzdkU-unsplash-1200-1.jpg","permalink":"/2019/07/azure-databricks-create-new-workspace-and-cluster/","title":"Azure Databricks - create new workspace and cluster"},{"content":"Another week and another round-up over data area. Take a look at what I have found for you last week.\nPress Azure DevOps - Build SSIS project (CI)\nLet\u0026rsquo;s build Continous Integration for SSIS from scratch.\nHow to Setup SQL Server Big Data Cluster in Azure (AKS) – Part 1\nMohammad (T) presents how to start with BDC.\nMapping Data Flow Parameters\nThis new feature allows you to pass parameters from Azure Data Factory Pipeline to its Mapping Data Flow.\nThe Recording for Using the Azure Data Factory Lookup and ForEach Activities\nAndy Leonard\u0026rsquo;s video about a useful scenario.\nWorkload replay with WorkloadTools\nGianluca (T) explains how to replay a Workload from production. Do make sure that you know what you looking for.\nSQL Server NOLOCK Anomalies, Issues and Inconsistencies\nAaron Bertrand (T) illustrates the hazards associated with the use of NOLOCK.\nQuick Tip: Power Query column by example\nMatthew Roche (T) shows briefly when Power BI can do the work for you.\nA word from (former) Microsoft’s CEO My friendship with Paul Allen changed my life. I loved sharing some of my favorite stories about him (and a few old photos!) at the Forbes Philanthropy Summit yesterday: https://t.co/3i6SFNg9q4 pic.twitter.com/t2t7yTBVie\n— Bill Gates (@BillGates) 28 June 2019\rSmile Corner ","date":"2019-06-30T20:38:51Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/06/last-week-reading-2019-06-30/","title":"Last Week Reading (2019-06-30)"},{"content":" Introduction Amit R S Bansal is a SQL Server Specialist at SQLMaestros (brand of eDominer Systems). He leads the SQL and BI practice with a much-focused team providing consulting, training and content development services to more than 160+ SQL customers globally. He frequently speaks at international conferences including Data Platform Summit (India), MS Ignite (US), SQLBits (UK), PASS Summit (US) \u0026amp; SQL Saturday events across the globe. Amit is also honoured with Microsoft Regional Director status, MVP award and Microsoft Certified Master (MCM) of SQL Server credential. Amit has been working with SQL Server since 1999 and has been part of countless mission-critical SQL deployments. Performance Tuning in SQL Server is his core area of expertise.\nThis talk has taken place during SQLBits 2019 in Manchester (UK) on 1st March 2019 (Friday).\nInterviewers: Kamil Nowinski (T), Michal Sadowski (T)\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript Kamil Nowinski: Hi, Amit! Thank you for accepting our invitation to the podcast “Ask SQL Family”.\nAmit Bansal: Thank you, thank you very much, it’s very kind of you to have me for this podcast.\nKN: Absolutely, it’s our pleasure. At the beginning could you tell us what your full name is and where you live.\nAB: My full name is Amit Bansal and I live in India. And in India, particularly I work out of two cities: Bangalore and Calcutta.\nKN: What do you do for a living?\nAB: I am running a company, basically as a director of a company by the name eDominer Systems Pvt. Ltd. and under which we have a couple of brands. I particularly take care of SQLMaestros, which is our brand related to doing a lot of work around SQL Server and Microsoft Data Platform, consulting and training and content development, and of course performance tuning assignments. And another part of my work is the community, which is data platform gigs.\nMichal Sadowski: How did you start working with SQL Server?\nAB: So the story about working with SQL Server goes back many years. I started my career as a developer. I used to do Visual Basic programming, and if I remember correctly it was Visual Basic 5, I think in 1997 or 1998 and we used to store our data in SQL Server 7. I did probably my first project with 6.5 but very quickly moved on to version 7.0. And in fact, in that year of 97-98 I had absolutely no knowledge of SQL Server. We just used to use SQL Server to store data. All we knew that there is a database there and there are some tables, and we used to just put data inside and take it out. And because we were all like application programmers doing a lot of front-end coding. But then, as our project size grew and data size started getting a little bigger, and of course there were some performance issues that were coming in, there was no one in our team who knew how to tune SQL Server databases or indexes.\nKN: No knowledge about indexes?\nAB: Yes, absolutely no knowledge about indexes. I didn’t even know what a B-tree structure is, to be honest, 20 years back. And yeah, then there was a good opportunity there and I took up the challenge to learn a little more about SQL Server. That’s where I think my formal training with SQL Server started. It was all self-learning. I picked up a few books and then books online and whatnot. And those days, you know, with SQL Server there were books online which used to be locally installed and you could read a lot more about SQL Server. That’s when it all started.\nKN: And what new steps did you make? You started reading those books, learning something and you took some courses, trainings, conferences maybe?\nAB: No, that was very early in my career to really even think about conferences. Honestly, in those years I didn’t even know what a conference is. As a matter of fact, because living in a city like Calcutta, and in those years India was not so tech-savvy and the IT story that we have in India today wasn’t there 20 years back. So I didn’t even know what a conference is. And there were no courses formally offered by companies out there. There were no such training institutes that would probably teach SQL Server, so it was all like learning by those books online which were locally installed when you installed SQL Server on your computer. No laptops though at that time, it was computers. So we read modules by modules and practice a lot of examples and then what indexes are, what B-tree structures are, how data is pulled out and reading a little more about relational engine and the query processing engine, etc. And that fascinated me so much that I left Visual Basic programming and I started focusing more on SQL Server.\nKN: So you started with Visual Basic? Which version?\nAB: Yes, Visual Basic 5.\nKN: Ha, I remember. I started with Visual Basic 6.\nMS: You present around the world and I would like to ask you about the Data Platform Summit conference. How does it looks compared to other international events?\nAB: So Data Platform Summit is very close to my heart. I of course conceptualized and architected the conference along with my team members. And how I would compare it is, it happens in India so of course the cultural differences are always there compared to US or compared to SQLBits happening in the UK and our conference happening in India. But apart from that, the overall format is the same. We have pre-cons, we have breakout sessions, so there’s so many things that are similar to begin with. The pre-con format, the breakout session format but we do a few additional things like we do chalk talks and open talks. We brought in these new learning formats, like chalk talks and open talks are like roundtable discussions, and chalk talks are no laptops, just whiteboarding and trying to explain a concept with the whiteboard and the chalk. That’s why we call this a chalk talk. So these new learning formats had been really very well appreciated by the community, and of course our conference is not as big as PASS Summit or SQLBits but we’re slowly getting there. Year by year it’s getting bigger. This year is the fifth year, last year in 2018 we kind of touched close to a thousand delegates for the conference, so hopefully we will make it bigger this year.\nKN: Nice. Also, you speak a lot, around the world. How do you prepare yourself for a speech?\nAB: So just like any other speaker, I would focus a lot on the content during preparation. Thankfully the audience calls me a natural speaker. I don’t take the credit for it. It is God-gifted, being a national speaker is a God gift so I’m very thankful to God that he made me a natural speaker so I don’t have to struggle a lot to get on to the stage to get to speak. But the content that you are going to present, that should be rehearsed. So I do spend time in preparing good content for my sessions. I focus less on PowerPoints because people really want to see things in action. And I believe content is king, so if people are spending\u0026hellip; for a pre-con let’s say people are spending a whole day with you, 8 hours, they really want a lot of rich content.\nKN: They expect something hot.\nAB: Absolutely, they expect a lot in pre-con. Especially when audiences pay extra for pre-cons, so they would really expect a lot. So a lot of preparation goes into pre-con, preparing all those demos and testing them again and they work fine. And because my specialization inside SQL is performance tuning, query tuning, troubleshooting, many of my demos are very unpredictable in nature, because I try to assimilate huge workloads with large databases. And they may not always for whatever reason run exactly the way I planned them to run. So lot of testing has to be done and I have to make sure that they run really well on that final day. So I spend a lot of time on demos and on the content. And for breakout sessions what happens is when you are used to spending a whole day, I mean when you rehearse a whole day’s content, then breakout sessions become a little easier. Because it is one hour or 75 minutes or like SQLBits, maybe 50 minutes, so it becomes a little easier.\nMS: So you start with a notebook, let’s say, and then you just write down the idea for the session, and then build around that all of the demos, all of the session, is it correct?\nAB: Yes, absolutely. So I’m taking down notes and what am I supposed to do step by step. And yeah, that’s pretty much it. I’m a little old school so I have a diary and a pen, so first I do that, like a rough sketch and then I formalize it, I use OneNote and I take my notes there.\nMS: What about last-minute changes? You are just before the session, and then some bright idea just came to your mind and then you think “OK, it will be worth presenting”. Do you include that in your presentation or do you try to avoid it?\nAB: I avoid doing any last-minute changes. It hasn’t really happened with me that I just get a very bright idea before the session. I’m constantly thinking about the existing content that I have to present, so it doesn’t really happen with me. But if at all it ever happened or it happens in future, I will not try to tweak the content on the fly so quickly, because that might disturb my flow. It may or may not work. I wouldn’t take that much risk.\nKN: So you had a pre-con here at SQLBits. It was yesterday or two days ago?\nAB: Yes, it was on Wednesday.\nKN: It was amazing, right? Could you tell us something about it? You had a lot of attendees on your session.\nAB: Yes, I had close to 70 people and thanks to SQLBits organizers for giving me an opportunity. This was my second time I was presenting a pre-con at SQLBits and the crowd was of course huge and the audience was wonderful. They were very patient and they interacted so well. That was amazing to see that everyone who came in the morning stayed till very end of the pre-con, which is wonderful. And something happened which was so unprecedented – I got great feedback on my LinkedIn when I posted it, so many people connected and they posted some wonderful comments. Yeah, so the pre-con went really well. Of course before delivering this pre-con here I’ve delivered it a couple of times earlier in other conferences, like Difinity and our own DPS and even SQLSaturday events. And in fact, this one day pre-con is actually an extract from my master class. And my master classes are like 40 hour, very intensive, deep dive, comprehensive training on performance and troubleshooting. So I took out the best parts, the best demos from there, the more relevant and common issues that people see, and I packaged them into this pre-con. So I think why I got such great feedback for this pre-con was because from the word go till the very end it was all demos. It was all demos. Even some concepts that I had to explain where I sometimes think a PowerPoint slide could have been better, I still did not use PowerPoint and I put those bullet points and notes as comments in my T-SQL file and store them in Management Studio. So people were just looking at code all the time, and there was demos, and I think that kept the entire session so alive and so intrigued by the demos.\nKN: It’s easy for you, you don’t have to switch between applications.\nAB: Yes, of course, it’s just double-clicking the SQL file and opening it in Management Studio.\nKN: I’ve visited your new website and on that website, in the menu, there are 3 blogs there. Which one is the most popular, or which one do you write on?\nAB: So I used to be a very prolific blogger some time back, so I used to\u0026hellip; Of course, I started writing SQL blogs first but then there were a couple of topics that I started writing about. A little bit about professional development, public speaking, presentation skills. And because I was traveling a lot, I started writing one or two blogs about travel tips. But I didn’t follow it very religiously and because of lot of travel and work, I was not able to maintain a consistent writing schedule and then my blog kind of slowly died down. Now a lot of people do visit blogs but they see that old content that is up there. I want to get back to writing again because that is still the most popular way of how people search for knowledge and constantly read on the go. You know, whenever they’re traveling and in their offices. But recently, I started recording lot of videos. Another thing which I saw, because my demos are so light and they were so helpful, so I thought “why not record short videos”. And I started recording them and I have put them in our YouTube channel, which is youtube.com/SQLMaestros and there are like bunch of videos. So whenever I get an opportunity, I record 10–15-minute videos of a SQL concept with a demo and then I put it up there. But yeah, doing video work here would not mean that blog is not important. I think blog still has a precedence over videos.\nKN: Video is just a new form of content. Probably it’s even easier for people to watch it.\nAB: Absolutely, it is easier, and because the internet is becoming so wide usage and with all those new smartphones and YouTube apps, watching videos has become very convenient. So a few years back, when you travel by train you would only see people with newspapers, and now there are no newspapers. People are with their phones and their headphones are on. And of course a few years back they had those headphones with wires, now they’re all Bluetooth and wireless. So yeah, video is becoming very, very popular. In fact, our channel, and I started it a few months back with zero, it is just touching 3000 subscribers now in just a short span of time. But yeah, but I still believe that a lot of community folks have given me feedback that “why am I not blogging”, even though they’re watching videos, but I really want to get back to blogging someday.\nKN: How much do you travel? Cause you mentioned it a lot. What do you mean by “a lot”?\nAB: So yeah, sometimes the travel schedule gets very, very bad. I mean it gets really crazy, so here is a very recent example: I started from Calcutta and went to Auckland via Singapore for the Difinity conference, and then after I had a pre-con and breakout sessions there, and from there I travelled to Wellington to participate in SQLSaturday Wellington, and Difinity was organized by Reza and SQLSaturday Wellington by Anupama, and I had pre-con and breakouts there. And meanwhile, my pre-con here in SQLBits was added in the second round, so I of course wanted to take this up as well, so I decided I will fly from Wellington to Manchester, so it was Wellington to Auckland and then Auckland to Doha and then Doha to Manchester. With all the stopovers and holds it was almost two days, almost 46 hours. Where, which timezone – God knows, but it was crazy travel. And then I’m going back to India now but then immediately after I go next few days, we all go to MVP Summit in Redmond. MVP and RD Summit. So yeah, so if you see the whole two months are all gone in travel. That’s how crazy it gets. But then after the MVP RD Summit there will be a bit of staying at home.\nKN: That must have a lot of impact on your private life.\nAB: It does, I’m very thankful to my family, my wife, my parents, my brothers, sisters and of course my kids. They cooperate so well, they support so well. I think without their support, it would have been difficult to really pull off doing so much work and travel. It’s nearly impossible. Family support is very critical.\nMS: What about your work-life balance? You have mentioned about your family, about the work, so how do you find the balance between these two?\nAB: I think it’s the same problem all of us face and nothing uncommon with me also here. Sometimes work really gets very hectic and family life does suffer in the role that you are, because we are just not doing a 9-6 job in our role because we run community, so people look up to us. We are community leaders, so when you become a leader and you are running communities, there is some kind of an implicit commitment that you have towards them. Apart from your day jobs, when you’re organizing free events and webinars, and then these events happen on Saturdays, right? And Saturdays-Sundays are supposed to be family time. So there’s a lot of this commitment that you have to fulfill and it comes in-between your family life. So there is that sacrifice there. But then it’s a passion that drives you and these are things that you do that make you happy. Like for example in India, when we do these Data Platform Day events which are very similar to SQLSaturday events of PASS, so we do these DPD events as we call it under data platform gigs and in an event you see like 200–300 people turning up. That amazing, that feeling of seeing so many community folks who are working on SQL and Microsoft Data Platform and then you deliver sessions to them and they talk about their problems and you network with them – that’s amazing. So you love doing that and you see that day happening and that keeps you driving.\nKN: Absolutely. Could you tell us about your MVP journey?\nAB: So I was awarded as MVP in the year 2007, so it has been now 12 years. And I think just like anyone else, I started SQL community activities in 2004. So it started from Calcutta where I did the first SQL user group meeting probably in India. I don’t know if that was the first one ever. And we had just seven people in that small user group meeting. And then it started from there and then of course one event got converted to two events, and then two to three, and when I say events it wasn’t really an event. It was just like a small meetup. There were a few people just coming together. And then it started getting a little bigger and because I was doing some training during that time along with Microsoft for SQL customers, then I used to travel from two different cities, like Bangalore, Bombay, Chennai, Hyderabad, these are pretty big cities in India and a lot of IT setup there. So whenever I used to do these assignments, Monday to Friday, I used to stay over for another extra day and organize these meetings on Saturdays. And it was just like word of mouth, because there was no social media at that point of time. There was no Twitter or\u0026hellip; I don’t even know if Facebook ever existed, then no LinkedIn, nothing, so it was just like word of mouth to Microsoft and a few customers “hey, we’re doing a group meeting here on SQL, why don’t you guys come over?”. It was just as informal as that. No formal registration.\nKN: It was hard to reach people.\nAB: Yeah, of course it was hard to reach people. Maybe we could just send across a few emails here and there, but that was pretty much it. And then that actually started getting a little bigger and better because something like that wasn’t really happening, but then of course who doesn’t want free training. So training, networking, get-togethers were getting really popular and Microsoft took a note of it and then one fine day I see that Microsoft has awarded me with the MVP status. I did get to know about MVP a few months back and I was like “wow this is a good thing to have” and then that eventually happened. And then of course after that, till now, till this, at least at this point while this interview is happening, I’m still an MVP. But yeah, you never know how things go in the future.\nKN: You absolutely deserved it.\nMS: What about your Microsoft Certified Master certification? It’s quite a huge challenge to get this.\nAB: All of you have done like good research!\nKN: Absolutely, we are prepared for this conversation.\nAB: OK, wonderful. So I got to learn about this MCM. I learned about MCM a little late in my so-called SQL journey. Because I think this certification was very popular and predominant in the United States and because I’m not in the US and I was in India, this certification was never popular there. And whatever, fortunately or unfortunately, I had never heard about it. But then, because of my frequent travel to US and working with the product teams also, and MVP Summit, somehow I got to know about this certification that they have something called MCM. And when I looked into MCM and I saw who MCMs are, and of course there were some really great folks in the community who were so popular and were so knowledgeable, they’re all MCMs. And I learned a little more about the certification process, that there is a knowledge exam and after that there is a practical exam, and that practical exam is going to be monitored by Redmond live for five hours and then you have to put that camera and whatnot. So all that really fascinated me but I did speak to a few folks in the community about this certification that you know, “is it really worth it?” because it was very expensive. I think it was like some two thousand - three thousand dollars and you know, whatnot. So it was very expensive so I was thinking that you know\u0026hellip; And of course that’s big money in India. So I was thinking “shall I go for it, is it worth it?”. Because there was no MCM in India. I think they were one or two folks but they worked for Microsoft and out of those folks who worked for Microsoft, also moved to Redmond. So there was literally in a way, back then, there was no MCM in India. And I was like “wow there is no MCM in India who is working outside Microsoft, I can be the first”, so that also excited me to some extent. And of course I got good feedback from the community about this program that it’s really worth it. Because sometimes many of these exams are more like multiple-choice questions and people read all dumps and whatnot and they just go and pass the exam. So I ever wanted to do that. But this MCM exam was very different. And yeah, the credibility factor, what I got from the community was that it is very credible and you should go for it. And then I decided to go for it. I started studying for it. I set up a group also, because a few people wanted to study along with me. I won’t take their names but they moved out of the group very soon. But I continued studying and I really prepared hard, I prepared well with all the resources that were there and I took the exam. I took the knowledge exam and then I passed and after that, I took the lab exam and then I passed the lab exam, and I got my MCM. And then I think at that time, my program manager for MCM was Bob Taylor, if I remember correctly. And then he put down on the blog about MCM and yeah, I was very happy. I remember, I was informed that I got my MCM credential on 31st December. I was in Bangalore at that time and my wife and I we all celebrated the New Year with an MCM. It was wonderful. I got, very proudly, like the first MCM of India outside Microsoft, so it was fantastic.\nKN: So how long did it take you to prepare?\nAB: Because there was so much work that was always going on, I did not take a break actually. It’s not like I put my work to rest, because the work is giving me my daily living, I can’t put it to rest and just study for an exam suddenly. So I was continuing to work and whatever time I was getting, I did spend extra time in preparing for the MCM exam. So I think I prepared for about two years.\nKN: Still long.\nAB: Yeah, because the amount of time I was able to spend was very little. So every day I would just spend very little and sometimes a whole week will go by and I would not study at all. So that’s why it was like sporadic and preparation\u0026hellip; slowly and slowly like a tortoise speed, for two and a half years I was covering module by module because Microsoft published a syllabus, like backup/restore, indexes and whatnot. So yeah, it took that much time. And then I was studying, studying, studying, preparing demos, I was seeing a lot of stuff and I was like “okay, I think it’s enough, let’s go and take the exam”. You know, one day I decided “okay, let’s go and take the exam”.\nKN: “I think I’m ready”, right?\nAB: That’s the best part. Because of working so much with SQL Server and then studying on top of it, I never felt I was ready. I was like “enough, let’s go and take the exam”. And I just went ahead with the exam.\nKN: Especially, as a professional, you work and get the experience\u0026hellip; It’s basically your work but you are still learning. That’s the good part.\nAB: Yes, absolutely, you’re always learning, you’re constantly learning. Every day you will learn something new.\nKN: So what is the personal achievement that you are most proud of?\nAB: Work-related or overall?\nMS: Up to you – it can be work, it can be personal, it can be both.\nAB: I think, touch wood, my God’s-graced, the personal achievement is that I am still with my family and I talk to them every day. We still live together, I can spend time with my parents and with my kids, I think that’s the biggest achievement above everything else.\nKN: How many kids do you have?\nAB: Two kids. My son is seven years old and my daughter is four years old.\nKN: Cool, wonderful age.\nMS: What hints would you give to young people that started to work on the IT market. They just graduated, they would like to start working with SQL Server.\nAB: First, young folks who are just graduating or about to graduate, it’s not that they should choose working with the product. SQL Server, at the end of the day, is a relational database software so it’s a product. They should focus on building careers. What they love doing the most. Many young folks, and when I say “young people”, I’m of course keeping the India perspective in mind. When they think about jobs and when they think about careers, they always go by which job or which career is going to pay me the most. Shall I be a data scientist, is that going to pay me the most? Shall I be a developer and then I can move all the way and become an architect and project manager, is that going to pay me a lot? I would advise them not to run behind the job that is going to pay the most, but what THEY love doing the most. So if they love playing with data, then they should pursue that. If let’s say things like machine learning algorithms excite them, then they should start thinking about building their careers in that path. Because there’s a saying: “if you make your hobby as a profession, you’re most likely to succeed”. So take examples of leading sportsmen: Roger Federer or in cricket you take name of Sachin Tendulkar. They pursued their hobbies and converted into profession.\nKN: So that kind of passion. Then you are happy with your work.\nAB: Absolutely. Because then you enjoy what you are doing. And you’re not just doing that for the sake of doing a job. Because when you start doing something just because it’s a job and you have to do it, then you wouldn’t enjoy and you wouldn’t really be very successful, at least in the long run.\nMS: What do you think about Microsoft’s certification program? Because it changed recently, so from the typical exams where you have 4 answers and then you’re just selecting one, into role-based certification.\nAB: I think it’s a good change that Microsoft has done. I got to know about this change just now when you told me. Because Microsoft certifications were something that I left long back. I think my last certification was in 2001, and in 2000 I did MCDBA and then MCSD .NET. After that, I have never done any Microsoft certification. Of course, MCM was different but then it was a different category altogether. So if they have done this change, it’s very good because those MCQs like multiple-choice question format wasn’t really taking a candidate anywhere. And as we know, the way the industry is, how people used to send proxies to do the exams and the internet was flooded with brain dumps and you know, cheat sheets and whatnot, and people just used to have all these paper certificates. You just call them paper certificates. So you would actually see a guy with 10 to 12 different Microsoft certifications but when he sits down for an interview, he doesn’t really know anything. And then you realize. So yeah, it’s good that Microsoft has changed that and made the exam more role-based so that you could test practical skills of the candidate. So I think that’s a great move.\nMS: Let’s talk a little bit about Azure Cloud etc. What do you think about the current trend of migration to the cloud. Is it going to replace SQL Server on-premise or not?\nAB: No, I don’t think Azure Cloud would replace SQL Server. I would say it will complement SQL Server. So going forward, you might see more hybrid deployments, but that notion that SQL Server will go away or on-prem will go away, that’s not going to happen. At least I don’t see that in the near future. But hybrid is a reality that we are seeing, where companies are looking at new projects being deployed on the cloud. Critical data is on-prem but then some portion of data is moving on to the cloud. There a lot of these scenarios and use cases, I think that is a reality. And cloud has its own advantages. It also has disadvantages. It’s not that it’s always a rosy picture out there. Companies have suffered, they have burned their fingers, we see cloud is very cost effective but sometimes we see that it is five times more expensive than what it could have been on-prem. So those realities are there and companies and vendors like Microsoft or Amazon they’re fighting it out to see how it can benefit every entity.\nMS: What do you think about the role of DBAs, database administrators within this hybrid environment? Is the role of DBA changing?\nAB: Of course, the role of DBA is changing and that’s for good because a DBA was only managing let’s say on-prem SQL Servers. Most customers that I worked with DBAs, there were of course a few cases where DBAs were like so hardcore with only SQL Server, but then with most companies, I was seeing DBAs are managing SQL Server as well as a bit of Windows infrastructure. But with the advent of cloud, we are seeing that now they are managing these deployments on the cloud, they’re managing their Azure accounts and credentials, and their sequel deployments in the cloud. So that’s an expansion and extension to their skill set. And this has to happen and it is happening very naturally and gradually. I think DBAs should be more open to expanding their skill set rather than to really put so-called a no-show to this change, that “I am a SQL Server DBA and I want to remain that”. No, the attitude should be that you have to move forward with the trend.\nMS: Until what point do you automate your environments? Are you a fan of automation?\nAB: Yes, of course. Automation is about improving productivity. So it’s not about being a fan or not being a fan. I think it is more about the need of organizations. Large organizations with let’s say dozens and hundreds of deployments, they really cannot survive without automation. Think about installing, think about patching, think about taking multiple backups of databases across hundreds of servers. Without automation it’s just humanly impossible to achieve that kind of resilience and consistency.\nMS: So once that’s forward, do you rely on automatic features like Plan Correction from Query Store?\nAB: At this point, maybe I’m just slightly not up to date with using those features right now because I’m still working with many customers who are in slightly older versions or the current versions, like 2012, 2014, etc. We’ve done a few POCs, where we just tried to see how new versions behave etc., but maybe I need to have a little more experience and see real-world usage of that. But yeah, from what I hear from the community that these features are very promising, so hopefully they’ll definitely do good. I have my faith in the SQL team, they’re bringing all these good features, so I’m sure they’re doing that with due diligence and of course they’re very customer focused, which is a great thing about the product team. There’s so customer-focused, so everything that they’re doing in their product is based on customers’ feedback.\nKN: For those customers that you are working with. Are they looking into migrating to Azure or how many of them are looking into such activities, in terms of percentage?\nAB: So I can put it this way: let’s say 20 per cent approximately are looking for immediate migrations, in the sense that they definitely want some portion of data to go on cloud, so they’re immediately ready to do something. I mean a hybrid scenario. Right now, I have no customer who says “no, I want to shut down my on-prem and I just want to go to the cloud”. I have no one who had ever told me that, and I don’t have any customer in that category. So if percentage is what you are looking for, then we have 20 per cent who are ready to migrate now. Migrate means the hybrid model. I could say that about 30 to 40 per cent of our customers who are thinking about doing something in the cloud, they are maybe doing POCs or they are still deciding their roadmap and their investments for the next year or so. And then the remaining are still dormant. And there may be government institutions where things move very, very slow, the adoption is very slow. So there are industries like that. They’re dormant, so they are just running with what they have, and they just know that there is something called cloud but what it is and how it works and things like that they just don’t know right now.\nMS: You have mentioned that you are learning every day, so what have you learned recently that was really interesting for you?\nAB: The most recent thing that I learned is that\u0026hellip; So we are doing this, in my demos I was doing these table-valued parameters. And with table-valued parameters, there are no statistics or indexes, with TVPs and TVFs. The cardinality estimator will always show you an estimated number of rows as one, but then I was trying to play around with recompile and I learned that “okay, with recompile you will still get some numbered, and in the new cardinality estimator, the estimation is hundred not one”. So yeah, these were a few new things that I just learned about when I was doing more research with the new cardinality estimator. That happened in recent pre-cons.\nMS: Do you have any list of the items that you would like to learn in the future?\nAB: Yes, in a summarized way, my learning list is of course to really go deep with some of the new features of SQL Server in the performance space. Query Store, Adaptive Query Processing, Intelligent Query Processing and all of that. I have overview of that because I’ve not dived very deep into it. And my diving deep is of different level altogether. So I want to spend quite a bit of time there with those things and upgrade myself there.\nMS: You have mentioned that not many of your clients are ready for the cloud but what do you think about the “cloud-first, cloud-only” approach?\nAB: “Cloud-first, cloud-only” approach I think is a statement that comes from Microsoft, from the product teams, because whenever they’re working on a new feature or designing something new, they think about cloud first. I think it’s good because they see that future is going towards cloud, so of course when you know that’s what the target is, then you will always keep that in mind. So I have nothing to criticize that approach. I think it’s good, but I think they should not forget the on-prem factor also, and I think it should be a balance of both.\nKN: Which part of your work is the hardest?\nMS: Is it travelling, delivering workshops?\nKN: Preparing for workshops?\nAB: Nothing is so hard as such. [laughs] I think the hardest part here is when too many things happen at the same time, because I have a team back in Calcutta, I have a team back in Bangalore and we’re doing community events also, and I’m traveling and doing conference sessions also. Then there is a masterclass coming up in March, we have to do some work for that also. Then there is some content recording going on. So when too many things happen. And then we are hiring new team members into our team. So when so many things pileup and are happening parallelly, then it gets a bit difficult to manage and coordinate.\nKN: So you need to manage your teams, right? One team or more teams? You mentioned at least two teams.\nAB: One team is in Calcutta, our Calcutta office, and the other team is in Bangalore, the Bangalore office. So there are two teams there, because we have two offices. So I manage both these teams.\nKN: How many members do your teams have?\nAB: My current SQLMaestros unit has about 20 team members.\nKN: Nice, good. OK, Amit. Thank you very much for being a guest on this podcast. It was a pleasure to have you.\nAB: Thank you very much for hosting me and inviting me for this interview. It took me by surprise but I’ve thoroughly enjoyed it and you made it so real. That’s the amazing part of it, so thank you very much.\nMS: Thank you. The very last question is about where we or the listeners can find you. Twitter, website?\nAB: Of course, just like anyone else, all channels are open. There is Twitter – @A_Bansal, that’s the Twitter handle. I’m there on LinkedIn as well. And what else? Of course I’m on Facebook. Probably the best is you go to Google, type Amit Bansal and I’ll show up. [laughs]\nKN: Thank you very much again.\nAB: Thank you, thank you.\nUseful links Amit R S Bansal - Twitter | Private Website | LinkedIn Profile| SQLMaestros Website |\nEvent: Data Platform Summit (India) | SQLBits (UK)\nPost Scriptum I may look pretty funny on the picture above actually. This is not a real suit what I have worn. SQLBits is also well known from Friday\u0026rsquo;s party and that was the time when we were conducting the conversation. So, it\u0026rsquo;s not my work suit - you see part of the party\u0026rsquo;s costume of mine.\n","date":"2019-06-28T06:00:09Z","image":"/uploads/2019/06/ASF023-SQLBits-with-Amit-Bansal_1200.jpg","permalink":"/2019/06/asf-023-amit-bansal-interview/","title":"ASF 023: Amit Bansal interview"},{"content":"In this blog post, I will answer the question I\u0026rsquo;ve been asked many times during my speeches about Azure Data Factory Mapping Data Flow, although the method described here can be applied to Azure Data Factory in general as MDF in just another type of object in Data Factory, so it\u0026rsquo;s a part of ADF automatically and as such would be deployed with other parts.\nWhenever I say Azure Data Factory in this post - I mean: Azure Data Factory v2. Forget about v1, ok?\nCode Maintenance From the very beginning, Azure Data Factory has the capability to keep the code of ADF synchronized with code repository.\nIn fact, when you hook up your ADF with the code repository, you\u0026rsquo;ll have two types of the instance of Data Factory:\nADF mode (the main one) - this is a deployed version of ADF GIT mode - separate version of ADF which is synchronized with repo and can be used to develop new features Two modes of ADF Both of these modes work differently. At the beginning after ADF creation, you have access only to \u0026ldquo;Data Factory\u0026rdquo; version. I described how to set up the code repository for newly-created or existing Data Factory in the post here: Setting up Code Repository for Azure Data Factory v2. I would recommend to set up a repo for ADF as soon as the new instance is created. Your life as a developer (also as whole developer team) would be much easier.\nWorking in \u0026ldquo;Data Factory\u0026rdquo; mode you can trigger any pipeline, so that will be visible in monitoring tool for everyone and available later on. When changing the code with UI, you can publish only fully validated code (whole ADF across all objects is being validated). This instance (code) is shared for everyone who has access to it. That also means you can lose your changes being overridden by someone else or you potentially could override someone\u0026rsquo;s changes when publishing the code as the latest one.\nIn Git repository mode (e.g. \u0026ldquo;Azure DevOps GIT\u0026rdquo;) you will be work on your own branch, so no changes lose risk, but also your development experience with ADF will be much pleasant. You don\u0026rsquo;t have to finish your work to be able to save the changes (yes - saving, not publishing), having the opportunity to test your version in debug mode. Bear in mind that in this mode you can not run your version of the pipeline through a trigger. The triggers run only deployed (\u0026ldquo;Data Factory\u0026rdquo;) version of pipelines. You would get the following warning once you try:\nADF code is JSON Before we go further, we must understand how the code of ADF does look like. Generally speaking, the code is kept in JSON format. The code reflecting the ADF in the repository is divided into folders and files. Folders represent object types, and each file represents one object in the ADF. So, under the main folder, you will see such subfolders:\ndataflow dataset integrationRuntime linkedService pipeline trigger Some of these folders might not exist when objects of particular types are not used in Azure Data Factory. Here is an example:\nAzure Data Factory and its JSON files in GIT code repository Comparison of code seen in ADF and GIT If we look inside the selected file - we will see exactly the same code that can be seen by working with Authoring of ADF (\u0026ldquo;Code\u0026rdquo; button in the right-upper corner of the screen when editing the object):\nAs you rightly notice - this approach is significantly different from the ARM Templates file structure, where we have everything in one, big file. Although it is possible to use these files for the deployment of ADF (Microsoft describes this here), I personally do not recommend this method.\nWhat\u0026rsquo;s all the noise about? You might be familiar with operations Import/Export ARM Template available from ADF. If you are not - nothing lost. If you are - good. Nevertheless, we will not use this option. Our goal is to automate the deployment process. No clicking, therefore, comes into play. We are going to use the best practice of working way with code repository with ADF, recommended by Microsoft.\nBefore we started working with ADF+GIT method - we had only one version of ADF (published) and an editing (non-saved) version. In that case, every time you want to save your changes, required to click the \u0026ldquo;Publish All\u0026rdquo; button. This is not a very convenient way to work with ADF, for several reasons:\nBefore writing, all objects in the ADF must be validated correctly Transitional changes (not fully finished) cannot be saved Your work can be lost or overwritten by another team member (the last publish counts) You simply can\u0026rsquo;t see a history of changes and ask for PR (Peer Review) Having ADF associated with (GIT) repository - you\u0026rsquo;ll see extra options in the top menu, as shown below:\nADF editor with GIT connected (bottom) and without (top) Once a connection to the GIT repository will be made - you can create your own branch (directly from ADF UI!) and all the above inconveniences disappear:\nSave your changes whenever you need without validation needs Avoid working on the same code with other developers Do follow the history of changes (each save do commit to the repo) Create Pull Request once your branch is ready to be merged to master (or collaboration) branch, which means: is ready to go! Switch between different version of the code (in a second) by toggling the branch! Boom! At the time of writing this post, working in the deployed version of ADF and having GIT set up - \u0026ldquo;Publish\u0026rdquo; button is available, however, it is not a recommended method of making changes in that ADF instance. You\u0026rsquo;ll be warned by following message (until 31/07/2019 when that option will be off):\nYou have GIT enabled in your data factory. Publishing outside GIT mode, when you have GIT enabled, will be disabled July 31st 2019. Please reach out to us using the feedback button above if you have any concerns. Read about our Best Practices.\nContinuous Integration and Continuous Deployment Since now, we are working on ADF associated with GIT repo and interested in automation the process of deployment our changes made to ADF. Right. How to do that if not by Export of ARM Template? At the moment, is very important to understand how ADF cooperate with GIT and distinguish action hidden behind buttons: Save All and Publish.\nAction: Save All This button once clicked, commits all the changes and pushes them to GIT repository into the branch you\u0026rsquo;re working with (seen on the left of \u0026ldquo;Save All\u0026rdquo; button). Check this out by reviewing the history of commits:\nCommit\u0026rsquo;s messages are generated automatically.\nAction: Publish The button can be used only from the (so-called) collaboration branch. Usually, the master branch plays this role.\nDuring this operation, two actions are being performed:\nPublishes the current version of code to \u0026ldquo;deployed\u0026rdquo; version of ADF New branch \u0026ldquo;adf_publish\u0026rdquo; is created (if not exist) and changes are being pushed to the branch Both operations can be found in the diagram below:\nAs you noticed - nothing extra is required to publish your changes to DEV environment (ADF Dev).\nAfterwards, a special branch has been prepared and this one should be used to make other actions in order to promote ADF to other environments.\nBear in mind that the code in \u0026ldquo;adf_publish\u0026rdquo; branch has got a totally different structure comparing to what we have seen in our branch:\nStructure of folders and files in adf_publish branch For the purpose of this post, we will be interested in only two files: ARMTemplateForFactory.json\nand the second: ARMTemplateParametersForFactory.json\nThe first contains all objects of ADF, while the latter contains values for parameters. You\u0026rsquo;re at home when familiar with ARM Templates, right?\nBuild Pipeline Now, we know how the structure of files in repo looks like and what we do need to deploy. Let\u0026rsquo;s focus on create build pipeline in Azure DevOps first. In Azure DevOps portal go to Pipelines and Builds, create a new build (New build pipeline).\nTo create an empty build pipeline - choose \u0026ldquo;Use the classic editor to create a pipeline without YAML.\u0026rdquo; option. In the first step, we must select the source code repository for this build. For this demo purposes, I use internal Git repo (Azure Repos Git), my \u0026ldquo;ADF-demo\u0026rdquo; repository. As default value for branch, you will see the \u0026ldquo;master\u0026rdquo; branch and usually, it is good, but not in this specific (ADF) case.\nicon-warning So, before click \u0026ldquo;Continue\u0026rdquo;, you must change source branch from (default) \u0026ldquo;master\u0026rdquo; to \u0026ldquo;adf_publish\u0026rdquo;. Then click \u0026ldquo;Continue\u0026rdquo;.\nAs we don\u0026rsquo;t want to use any template - click \u0026ldquo;Empty job\u0026rdquo; in order to go to further step.\nNow, let\u0026rsquo;s configure some parameters for each section:\n(Build) Pipeline Do name your build pipeline and select agent who does the job. You can use one of the agents offered by the portal or yours one if you have got.\nAzure DevOps: Build pipeline Get sources\nThis section holds the pointer to the code repository to collect the code and it contains all the information you have provided in the previous step (Team project, Repository, Default branch). Again: it\u0026rsquo;s important: check if you have selected \u0026ldquo;adf_publish\u0026rdquo; as a source branch.\nAgent job 1\nThis is the section when we might configure our requirements for the agent. As we are not very demanding here - you can leave all options by default.\nAim of a build process is to build a project and generate artifacts which will be used for deployment. As there is nothing to build here (ADF validates all object while publishing) - all we need to do here is simple copy files = publish artifact. Hence it\u0026rsquo;s time to add a new step for the agent. Click on \u0026ldquo;+\u0026rdquo; (plus) button onto \u0026ldquo;Agent job 1\u0026rdquo; and find \u0026ldquo;Publish Build Artifacts\u0026rdquo; task.\nAzure DevOps: Adding a new task Add a new task and configure it as shown below replacing \u0026ldquo;Path to publish\u0026rdquo; with your one:\nAzure DevOps: Configure Publish Artifact task That\u0026rsquo;s all to build pipeline. Now you can Save \u0026amp; queue it in order to create your first ADF build.\nRelease Pipeline Once we have some artifacts prepared by build pipeline - we are ready to go with a deployment. Let\u0026rsquo;s create a new Release Pipeline, again without using any template (Empty job).\nAzure DevOps: Creating a new release pipeline Then, for Artifacts - click Add and configure Artifacts that have been created in Build Release phrase. Afterwards, do configure the first Stage (know as well as environments) calling it \u0026ldquo;UAT\u0026rdquo;. The next step is to add two tasks:\nTasks ADF instance deployment This is Inline script of Azure PowerShell type of task which creates ADF for us if that does not exist.\nAzure DevOps: Instance of ADF deployment task As you can see, I have used one parameter and hard-coded values for Resource Group name and Location. Both can (and should!) be replaced with variables.\nADF all object deployment This step is for actual deployment of all ADF objects like:\nPipelines Linked Services Datasets Data Flows Triggers Just add new task type \u0026ldquo;Azure Resource Group Deployment\u0026rdquo; and configure all fields as shown below:\nAzure DevOps: Deployment of all ADF resources (objects) Field Value Action Create or update resource group Resource group Your name of RG created in the previous task in this pipeline Location Location of your ADF Template location Linked artifact Template $(System.DefaultWorkingDirectory)/_DevOps-AdfDemo-CI/drop/ARMTemplateForFactory.json Template parameters $(System.DefaultWorkingDirectory)/_DevOps-AdfDemo-CI/drop/ARMTemplateParametersForFactory.json Override template parameters -factoryName $(FactoryName) Deployment mode Incremental In your example, you must replace the build name, which is _DevOps-AdfDemo-CI in my demo. You also might have noticed that \u0026ldquo;drop\u0026rdquo; is an artifact name defined in Build Pipeline.\nAnother important parameter is \u0026ldquo;Override template parameters\u0026rdquo;. In that field, we can define all parameter values that should be replaced. It\u0026rsquo;s crucial to provide all required values that differ between environments otherwise values from template parameters would be used. Although it\u0026rsquo;s not the best method of replacing parameter values - in this post I wanted to simplify the process as much as possible.\nLast but not least parameter is \u0026ldquo;Deployment mode\u0026rdquo;. You can select between \u0026ldquo;Validation Only\u0026rdquo;, \u0026ldquo;Complete\u0026rdquo; or \u0026ldquo;Incremental\u0026rdquo;. Even though we point out the complete version of ADF template (file) - Incremental mode must be selected here. Complete would remove the entire ADF instance, pretend that deploy everything else and finish without any error, but your ADF instance just\u0026hellip; gone (has been deleted!). Check for yourself or watch this video where I documented out that behaviour (2'14\u0026quot; is the moment when ADF created a few seconds ago disappears):\nhttps://www.youtube.com/watch?v=ZzvwaP5JOE4\nSo, remember then: \u0026ldquo;Incremental\u0026rdquo; mode is the only one correct option here.\nPS. Yes, I know, the documentation said so, however it still strange behaviour to me.\nVariables If you\u0026rsquo;re not familiar with Azure DevOps very well - the Variables can be defined in Variable tab. Very briefly: they might be defined providing the values directly, defined for specific stage and also they can be provided by Azure Key Vault reading the values from its secrets. In this demo I\u0026rsquo;ve used only one variable, so you must define at least FactoryName.\nRun, run release! Once you finish creating a new Release Pipeline in the portal (don\u0026rsquo;t forget to save it) - it\u0026rsquo;s time to create new/first Release and deploy the code of ADF with fully-automated way. Do click on \u0026ldquo;Create release\u0026rdquo; and \u0026ldquo;Create\u0026rdquo; button in a new window.\nNext step As an extension to the above pipeline, you can clone the existing Stage (UAT) to prepare deployment for other environments, such as PreProd \u0026amp; Prod. If you do so, don\u0026rsquo;t forget to close variable(s) with correct values for these stages.\nAdding Pre-Deployment approvals allow you control who/when will be able to deploy ADF with DevOps. This configuration, however, goes beyond the scope of this article.\nQuestions Can I change the Git repository when my ADF is already hooked up to a different one?\nYes, you can. To achieve that you must disconnect ADF from current GIT repo and then hook up another one.\nWhat code repositories are being supported by ADF now?\nCurrently, you can connect ADF to the following types of repositories:\nGitHub Azure DevOps Git Conclusion CI/CD process for ADF doesn\u0026rsquo;t similar to the others you might know. To be fair, it\u0026rsquo;s little intuitive when you look at it at the beginning. That\u0026rsquo;s why I tried to describe it as straightforward as possible, bearing in mind, that the post does not cover more complex scenarios. However, I believe that is a good start kit, but obviously, the best method of verifying is to give it a try yourself. In the next post of this series, we\u0026rsquo;ll take a closer look the way of passing parameters values in a more efficient and comfortable way.\nBecause many people asked me (including myself a few months ago) how to set up that process, many of us still have a mess in their head.\nAnd because the process is relatively new - not too many have an idea of how to start with it. I hope this post has helped you at least a little bit to understand the process. Let me know if you have whatever questions about this and I will try to help.\nMore sources - links Azure DevOps\nAzure Friday: Continuous integration and deployment using Azure Data Factory\nVisual authoring in Azure Data Factory\nContinuous integration and delivery (CI/CD) in Azure Data Factory\nContinuous integration and deployment using Data Factory\n","date":"2019-06-26T06:00:08Z","image":"/uploads/2019/06/steel-1968194_1920.jpg","permalink":"/2019/06/deployment-of-azure-data-factory-with-azure-devops/","title":"Deployment of Azure Data Factory with Azure DevOps"},{"content":"Hello folks! Let\u0026rsquo;s take a look at a set of information from the field again.\nPress The best practices for performance comparison between Azure SQL Managed Instance and SQL Server\nJovan Popovic describes 3 stages when doing a performance comparison between SQL Server and Azure SQL MI.\nSQL Injection Protection\nStill reported as the second most common technique for web attacks.\nAzure Data Factory - Web Hook vs Web Activity\nA good comparison is done by Paul Andrew (T).\nMinimal Logging with INSERT…SELECT and Fast Load Context\nDid you know that not always fully logged code path is being applied? Very good article by Paul White (T) with code examples.\nWhy You Should Create Reports in Power BI Desktop Instead of the Power BI Service\nMelissa (T) describes a scenario when the editor in Service is not quite a good choice.\nGlobally Distributed Azure Cosmos DB\nA few words about benefits and a few steps about how to create it.\nServerless geo-distributed applications with Azure Cosmos DB (video)\nMatias Quaranta (T) joins Scott Hanselman (T) to share some best practices for creating serverless geo-distributed applications with Azure Cosmos DB.\nDoc: Azure DevOps - Logging commands\nSpecific formatting or (progress) action can be achieved.\n30 days of DevOps: Application Logging\nA test of few applications who help you to check if a deployed application working as expected.\nA word from Microsoft’s CEO I chatted last week with @MarriottIntl CEO Arne Sorenson about how digital technology is changing the hospitality industry, transforming how Marriott interacts both with its customers and its associates. Here’s our conversation. https://t.co/RHaKBK91S2\n— Satya Nadella (@satyanadella) 18 June 2019\nVideo: Airbus drives innovation with Azure mixed reality and HoloLens 2 Smile Corner ","date":"2019-06-23T18:27:18Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/06/last-week-reading-2019-06-23/","title":"Last Week Reading (2019-06-23)"},{"content":"\nI\u0026rsquo;m happy to announce that the next episode of \u0026ldquo;Ask SQL Family\u0026rdquo; podcast is coming.\nThis time our guest is Amit Bansal. We were talking to him during this year\u0026rsquo;s SQLBits conference.\nAre you interested in listening to an interview with the only one MCM in India?\nSimply click the PLAY button above or move to the YouTube: https://youtu.be/fM4OnMVMO_A\nEnjoy!\n","date":"2019-06-21T06:00:46Z","image":"/uploads/2019/06/img_5d0c0f30f01ac.png","permalink":"/2019/06/asf-023-video-invitation/","title":"ASF 023 video invitation"},{"content":"It is the fifth post on series related to installation and configuration of SQL Server client tools required.\nThis series is for professionals who start their journey with SQL Server administration and also for those who want to extend and structure their knowledge on SQL Server administration.\nPreparations We are starting with the following assumptions:\nAll steps described in this first blog post has been configured ISO file with SQL Server binaries and Cumulative Update files are available - steps how to download it are described in the second blog post There is a connection between Windows server and domain controller No issues with connectivity with Active Directory domain (no AD-related errors in System Event Log) SQL Server instance is installed as described in the second blog post or third blog post Management tools are installed as described on forth blog post Configuration using SQL Server Configuration Manager Start SQL Server Configuration Manager and browse to SQL Server Network Configuration node and select Protocols for MSSQLSERVER\nBy default only Shared Memory option is enabled thus access to SQL Server is possible only from local machine. To enable remote access to SQL Server you need to change TCP/IP option by right-clicking and selecting Enable option.\nAlternatively, you can go to properties and change Enabled option from No to Yes.\nAs a second step, go to \u0026ldquo;IP Addresses\u0026rdquo; tab and scroll down to \u0026ldquo;TCP Port\u0026rdquo; option under IPAll section. SQL Server is using well-known port 1433, however, it is advised to use other than the default port. It will protect you from automated tools that scan a network for appearance of SQL Server.\nSave changes and go to SQL Server Service and check if SQL Server Engine and SQL Server Agent are configured to have Automatic Startup Type.\nSQL Server Browser should be left Disabled.\nAs the last step in SQL Server Configuration Manager, you can set trace flags to be enabled during start of the SQL Server services. Below there is a list of the trace flags that can be considered to be used on Production use:\nT1117 T1118 T4199 More information about trace flags you can find on the following link.\nReview of the instance after installation using SQL Server Management Studio Checking Service Principal Names As a first step, we are going to connect to SQL Server instance using SQL Server Management Studio.\nStart SQL Server Management Studio and connect to the SQL Server Instance (local server in our case).\nGo to instance name, Management, SQL Server Logs and select first one from the list (current one):\nReview all SQL Server Log entries and note if there is anything with warning or error message.\nOne of the most common issues that can be found after installation is an issue with registering Service Principal Name for SQL Server Engine account. The message in the SQL Server Log is as follows:\nSQL Server is attempting to register a Service Principal Name (SPN) for the SQL Server service. Kerberos authentication will not be possible until a SPN is registered for the SQL Server service. This is an informational message. No user action is required. The SQL Server Network Interface library could not register the Service Principal Name (SPN) [ MSSQLSvc/WINSQL.datacommunity.local ] for the SQL Server service. Windows return code: 0x200b, state: 15. Failure to register a SPN might cause integrated authentication to use NTLM instead of Kerberos. This is an informational message. Further action is only required if Kerberos authentication is required by authentication policies and if the SPN has not been manually registered. The SQL Server Network Interface library could not register the Service Principal Name (SPN) [ MSSQLSvc/WINSQL.datacommunity.local:1433 ] for the SQL Server service. Windows return code: 0x200b, state: 15. Failure to register a SPN might cause integrated authentication to use NTLM instead of Kerberos. This is an informational message. Further action is only required if Kerberos authentication is required by authentication policies and if the SPN has not been manually registered. To fix this issue there is a need of registering SPN for SQL Server Engine by taking values from the SQL Server Log messages and running setspn command (it requires domain admin permissions)\nsetspn -A MSSQLSvc/\u0026lt;hostname\u0026gt;:1433 \u0026lt;SQL Server Engine account\u0026gt; setspn -A MSSQLSvc/\u0026lt;hostname\u0026gt; \u0026lt;SQL Server Engine account\u0026gt; In our case commands looks as follows:\nsetspn -A MSSQLSvc/WINSQL.datacommunity.local:1433 DATACOMMUNITY\\SQLServerEngine$ setspn -A MSSQLSvc/WINSQL.datacommunity.local DATACOMMUNITY\\SQLServerEngine$ To list all SPNs registered for specific service account you can use the following command:\nsetspn -L DATACOMMUNITY\\SQLServerEngine$ The output should like as follows:\nAfter registering SPNs restart of SQL Server service is required:\nCheck SQL Server Log after the restart - SPNs should be registered successfully.\nNote: SPNs and communication using Kerberos authentication is critical when using SQL Server features like SQL Server Reporting Services, AlwaysOn Availability Groups, etc. Make sure that SQL Server Log is clear from any issues and SPNs are configured correctly.\nInstance configuration using SQL Server Management Studio Now let\u0026rsquo;s move the configuration of the Instance.\nRight click on the instance name and go to Properties:\nGo to Memory and modify Minimum and Maximum Server Memory to match the following formula:\n- Maximum Server Memory = Amount of RAM on your machine - (minus) 4GB (for Operating System)\n- Minimum Server Memory = half of the value of Maximum Server Memory\nExample: Server has 64 GB of RAM, thus Maximum Server Memory should be 60 GB (61440 MB) and Minimum Server Memory should be set to 30 GB (30720 MB).\nGo to Security and check if Windows Authentication mode is selected. This authentication mode is safer than SQL Server authentication mode thus is advised to be used on a Production system.\nGo to Database Settings and enable Compress backup and Backup checksum options. Validate if paths provided in Database default locations are correct\nGo to Advanced and modify Cost threshold for Parallelism to value 50 and Max Degree of Parallelism to number of processors on your machine but not more than 8\nYou can close Server Properties window by clicking on OK - your server has been configured for better performance.\nHowever, not all options are available in the Graphic User Interface and configuring more than one server with this approach can be tedious task thus there is another better way of doing it: using T-SQL commands described in next section.\nReconfiguration using T-SQL As a first step, we are going to connect to SQL Server instance using sqlcmd program. It is command line program thus you need to start cmd\nOnce cmd is started check if it is running with administrators permissions:\nNow you can run sqlcmd command with correct arguments. For the list of arguments run:\nsqlcmd -? You should have a list of all arguments that can be used for the sqlcmd:\nConnect to the SQL Server located on the same machine by running following command\nsqlcmd -S. Note: option -S specifies a server, in our case, it is local server so \u0026ldquo;.\u0026rdquo; is used. You can also use (local)\nand running following T-SQL command to list configuration options of the SQL Server instance:\nEXEC sp_configure; GO There should be following the output of the query:\nNote that show advanced options is set to 0, so not showing all available options.\nTo change this behaviour run following command:\nEXEC sp_configure \u0026#39;show advanced options\u0026#39;,1; GO The output of the query should contain much more entries than before:\nThe interesting options for us are options for modifying the following options:\nbackup checksum backup compression cost threshold for parallelism Database Mail XPs max server memory min server memory max degree of parallelism remote admin connections Note: For Remote Admin Connections TCP Port 1434 needs to whitelisted on the server where SQL Server instance is installed.\nFollowing T-SQL statement modifies above settings:\nEXEC sp_configure \u0026#39;backup checksum default\u0026#39;,1 EXEC sp_configure \u0026#39;backup compression default\u0026#39;,1 EXEC sp_configure \u0026#39;cost threshold for parallelism\u0026#39;,50 EXEC sp_configure \u0026#39;Database Mail XPs\u0026#39;,1 EXEC sp_configure \u0026#39;max server memory (MB)\u0026#39;,8192 EXEC sp_configure \u0026#39;min server memory (MB)\u0026#39;,4096 EXEC sp_configure \u0026#39;max degree of parallelism\u0026#39;,4 EXEC sp_configure \u0026#39;remote admin connections\u0026#39;,1 EXEC sp_configure \u0026#39;show advanced options\u0026#39;,0; RECONFIGURE GO Below screenshot with all commands and the output:\nYou have successfully configured the SQL Server Instance.\nSummary Configuration of the SQL Server instance is one of the most important steps after installation, that often is overlooked. This blog post describes how to configure the instance using both SQL Server Management Studio and sqlcmd program. The latter option can be used in environments where more than one SQL Server is installed in a time efficient way.\nLinks More information about steps described above can be found on the following sites:\nSQL Server Configuration Manager\nSQL Server Trace Flags\nRegister a Service Principal Name for Kerberos Connections\nServer Memory Server Configuration Options\nConfigure the cost threshold for parallelism Server Configuration Option\nConfigure the max degree of parallelism Server Configuration Option\nsqlcmd Utility\n","date":"2019-06-18T08:00:23Z","image":"/uploads/2019/06/electric-1080585_1280.jpg","permalink":"/2019/06/configuration-of-sql-server-instance/","title":"Configuration of SQL Server instance"},{"content":"Hello Data girls and guys! A lot of Microsoft\u0026rsquo;s announcements in many areas of the data platform. Here you have a big portion of news today.\nPress Unlock the value of data faster through Modern Data Warehousing\nHow long it takes to unlock the value, the value of data? Simon (T) explains.\nAzure SQL Database Edge: Enabling intelligent data at the edge\nA new dimension of the \u0026ldquo;edge\u0026rdquo; concept.\nHow to create and use App Navigation in Power BI.\nDo understand how to build a well-tweaked PDF pipeline.\nSQL For Cosmos DB – Handling Complex JSON Structures\nAdam Aspin reviews the Azure Cosmos DB SQL API from the perspective of the relational database developer.\nAnnouncing New AI and Enterprise features for Power BI\nIn this summary, you can read about new AI function, data flow and capabilities in Power BI.\nWhat\u0026rsquo;s new and planned for business intelligence\nPlan for the next half year.\nSnowflake Is Now Available on Microsoft Azure Government\nCool. Now, we are waiting for this in other regions.\nMicrosoft Business Application Summit\nHave you missed MBAS? Do not worry! All the video records are available for you!\nGeneral availability: 4 vCore Azure SQL Database managed instances on Gen5 hardware\nPeople voted: Make smaller instance sizes available. Task accomplished.\nBlocked Process Detection and Analysis\nHmmm, I did not even know about that threshold before.\nA word from Microsoft’s CEO A fantastic example of the power of AI to help an organization maximize the value of its most important asset – its data – to optimize operations and better serve customers. https://t.co/wxbuM3UCfo\n— Satya Nadella (@satyanadella) 13 June 2019\rVideo: Microsoft Power BI The future of modern BI - roadmap and vision\nSmile Corner ","date":"2019-06-16T22:03:38Z","image":"/uploads/2019/05/news-absolutvision-365898-unsplash-390.jpg","permalink":"/2019/06/last-week-reading-2019-06-16/","title":"Last Week Reading (2019-06-16)"},{"content":"In this post, I will show you one of the scenarios when SSDT (SQL Server Data Tools) can fail unexpectedly while publishing database project to target server or preparation of changes script (Generate script), which is actually the first part or publish action.\nScenario I had a solution with a few database projects. Visual Studio 2017 (Version 15.7.6) with SSDT (15.1.61901.09210). We were working on this project on and off making small changes in various databases. Not too much work, but then the deployment needs to be done. Having few environments (i.e. DEV, CI, UAT, PrePROD, PROD) it\u0026rsquo;s a bit challenging (and wrong!) to do this manually, even though with (PowerShell) script.\nThe best option is to prepare a fully automated deployment process in Azure DevOps. So, I did. That one was pretty simple, without additional tests, reports, etc. Baby steps, right? Just simple build pipeline and release pipeline for all databases in the solution in order to replace manual work with automating one.\nOnce I have done - build works perfectly, release\u0026hellip; almost. Release pipeline contains several steps - one per database. Job executed only partially and stopped at one database:\nLet\u0026rsquo;s double check the details:\nThat\u0026rsquo;s the error developers really do not like: saying that something is wrong\u0026hellip; where SOMETHING is a keyword. Nothing else. No hints, no details.\nHence, I decided to return to the roots - Visual Studio. Solution rebuild - fine. When I was trying to deploy (publish) THAT database to THAT server, I\u0026rsquo;ve got the following error:\nObject reference not set to an instance of an object.\nThis is a bug in SqlPackage.exe as the error is unhandled and that\u0026rsquo;s the reason why the process looks like running in Visual Studio (\u0026lsquo;Data Tools Operations\u0026rsquo; tab), but really - the thread under the hood crashed.\nI started to investigate it deeper and the facts I found are the following:\nAll databases from the solution exist on the server All other databases in the solution were capable to publish to THAT server - OK THAT database causes the error when trying to publish to THAT server - error Publish THAT database to ANOTHER server (environment) - works fine Publish THAT database to THAT server, but NEW database - works fine Cleaning solution (menu Build -\u0026gt; Clean solution) and rebuild doesn\u0026rsquo;t help - still the same error Closing Visual Studio, deleting *.dbmdl files for the database(s) didn\u0026rsquo;t help - still the same error It doesn\u0026rsquo;t matter whether you use a publish profile to deploy DB or publish action (right click on database node -\u0026gt; Publish) - still the same error All factors indicate that the target server and other databases are not guilty. The problem only occurs with this particular database (and a specific server) when we try to generate a differential script. So the problem has to be somewhere in the differences between the project and the database on the server.\nAfter a few hours of investigation, I found the exact differences, the exact SET OF OBJECTS that cause the problem.\nCircumstances The error happens in the case when:\nDatabase project contained a table with partitions (schema+function) The table is already deployed on a target server (database) The table is excluded in SSDT (but schema+function are remaining) You try to deploy DB without \u0026ldquo;deleting objects not exist in the source\u0026rdquo; (means the option is OFF) Simply - SSDT can\u0026rsquo;t cope with that scenario and a thread crashes.\nSolution/workaround Interesting enough - all above must occurs. So, the workaround of this issue is \u0026ldquo;easy\u0026rdquo; (when you know what need to do - it\u0026rsquo;s often easy).\nDo one of the following actions:\nExclude partitions for that table from the project (not recommended if the table should exist on the database) Drop the table from target database (recommended if you don\u0026rsquo;t need it. In that case - do above action as well) Set option \u0026ldquo;deleting objects not exist in the source\u0026rdquo; (always recommended, but be careful with that) Include the excluded table in the database project I agree that the described case is pretty unusual, but in our often-changing environments and code - that might happen. SSDT is a very good tool that no other databases have and generally speaking it is very useful for building database solutions. Unfortunately, when you found yourself in such a situation like this - you will waste a lot of time. The worst thing I really don\u0026rsquo;t like in such situations is lack of tip(s) in the error message. Would be good to have at least a small hint that could lead you in the right direction.\nThanks for reading!\nUpdated @ 08/11/2019\nThis post is already part of the whole series about SSDT.\n","date":"2019-06-12T07:30:03Z","image":"/uploads/2019/06/ssdt-crash-when-publishing.jpg","permalink":"/2019/06/ssdt-sqlpackage-thread-crashes-with-error-when-publishing/","title":"SSDT (SqlPackage) thread crashes with error when publishing"},{"content":"Hello there! Another week, another roundup of Internet interesting stuff.\nPress Microsoft Business Applications Summit\nThe event is happening now in the Georgia World Congress Center in Atlanta, GA. Keynote will be available online.\nPreparing for and Taking Microsoft Exam DP-200 (Implementing an Azure Data Solution)\nCathrine (T) presents tons of useful links and tricks when you learn to that exam.\nWhat You Need to Know About Data Classifications in Azure SQL Data Warehouse\nMeagan Longoria\u0026rsquo;s (T) first glance and scenario when it doesn\u0026rsquo;t work.\nDatabase Delivery with Docker and SQL Change Automation\nPhil Factor (T) demonstrates the basics of how to automate database builds into a Linux SQL Server container running on Windows.\nWhy is Sqlpackage Using All The Build Server Memory?\nRichie Lee (T) takes a closer look at the performance of sqlpackage.\nAzure Databricks and CosmosDB: Tips on playing nice together\nCase: Leverage Azure Databricks to ingest data from CosmosDB.\nCHOOSE() in SQL Server\nThere is a small chance that you don\u0026rsquo;t know this function.\nA word from Microsoft’s CEO Our @MSMakeCode team is working to empower students to not just imagine the future but to build it – providing them with a free, open source learn to code platform. Take a look. https://t.co/sxpmR5Q2EH\n— Satya Nadella (@satyanadella) 6 June 2019 Video: SQL Server on Linux https://www.youtube.com/watch?v=S-EVMQKoXck\nSmile Corner ","date":"2019-06-10T07:37:07Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/06/last-week-reading-2019-06-09/","title":"Last Week Reading (2019-06-09)"},{"content":"Hello data folks. Here you have some updates from the data world.\nPress Where should I clean my data?\nJames Serra\u0026rsquo;s (T) guidance when using ADF, DW, maybe Databricks\u0026hellip;\nHow To Trigger A Power BI Dataset Refresh Using An Azure Data Factory Web Activity\nDave Ruijter (T) shows secure with Azure Key Vault step-by-step recipe.\nGetting started with Azure Data Factory Mapping Data Flows\nMatt How (T) describes MDF with a simple example - read it if you don\u0026rsquo;t know Mapping Data Flow yet.\nUsing Azure Data Factory Mapping Data Flows to populate Data Vault\nAnother loading pattern with ADF.\nScripting out SQL Server Data as Insert statements via PowerShell\nPhil (T) built a script for you - script your data to native BCP format either.\nUpgrading a data model from Power BI to Tabular got a lot harder\nAlthough it\u0026rsquo;s not supported by Microsoft, worth to know the trick, especially if you do it to an earlier than 2019 version.\nASF 022: Guy in a Cube interview (part 2)\nHave you had a chance to listen to the latest episode with Adam and Patrick?\nA word from Microsoft’s CEO Great to be visiting Sweden today, where we're announcing our ambition to build datacenters in the country that will be among the most sustainable in the world. https://t.co/zHE3nf7cwn\n— Satya Nadella (@satyanadella) 29 May 2019 Video: Self-driving cars https://www.youtube.com/watch?v=ugNJJf2QW0E\nSmile Corner ","date":"2019-06-02T20:30:34Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/06/last-week-reading-2019-06-02/","title":"Last Week Reading (2019-06-02)"},{"content":" This is the second part of the conversation.\nIf you are not familiar with the first part - you can start from this post.\nIntroduction Guy in a Cube is all about helping you master business analytics on the Microsoft Business analytics stack to allow you to drive business growth.\nThey are just two guys that do the work.\nAdam \u0026amp; Patrick look at how to leverage Microsoft Business Analytics to allow you to gain knowledge that is needed to shape the data your business cares about. This includes Power BI, Reporting Services, Analysis Services and Excel. If you work with our business analytics products or services, be sure to subscribe and join in the discussion with their weekly content on YouTube channel.\nAdam Saxton and Patrick LeBlanc are Microsoft employees.\nThis talk has taken place during SQLBits 2019 in Manchester (UK) on 28th February 2019 (Thursday).\nInterviewers: Prathy Kamasani (T), Kamil Nowinski (T).\nHow much extra commitment helps at work and what else?\nIn what position did Patrick no good and why he was sweating?\nWhat is tremendously difficult for Patrick, but is child\u0026rsquo;s play for Adam?\nWho is Adam\u0026rsquo;s mentor and how has he started speaking on workshops?\nIn this episode is predominantly about commitment, youtube, extra efforts, moments of discouragement, how to do things well and how to start.\nThere will be a few things about Star Wars, Power BI and Microsoft teams, Kerberos and DTS too.\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript Prathy Kamasani: I think this is, … I don’t know. I personally want to ask if… I don’t want\u0026hellip; So doing the YouTube videos\u0026hellip;\nAS: If you can ask, it’s gonna be all the time.\nPK: I can cut it. Whatever do you guys do with YouTube, obviously it’s helping you learn the technology, but is it helping you career-wise?\nAdam Saxton: It helps in many ways.\nPatrick LeBlanc: Oh, man.\nAS: It helps in so many ways.\nPL: Oh, oh, oh.\nAS: It’s not just career, it’s not just technology, it’s\u0026hellip;\nPL: From a career perspective I will tell you this. I wasn’t in the product group. I was in sales. And the people, the sales guys that I worked with, they would say ‘Hey, Patrick, okay so we’re going to work with\u0026hellip;’ and I worked in education or I worked in state and local and they’d say ‘Hey, Patrick, we’re gonna go and talk to such and such. But you don’t come in the room right away. You can hang out, sit in the hall, and we’ll do it, and when we’re ready for you, we want you to come in’. And then I would walk in and people will go: ‘Do you know who that is?’ Right? ‘Oh my God, do you know who that is?!’ Like that, right? And they would use me as\u0026hellip; and my manager, so my managers\u0026hellip; It made them look good, right? There was a contest at Microsoft and they wanted you to tell a story about something you did.\nAS: It had to be a video.\nPL: Had to be video. So I had to make a video on this thing that I created. I created this Power BI at-risk model. It helped teachers identify at-risk student’s. And so I built this out, and so I call Adam up when I was like ‘Dude, I want to do a video. I’m gonna edit it\u0026hellip;’\nAS: No, no, what he said is like: ‘I need you to teach me how to use Camtasia’.\n[laughter]\nAS: And I’m like: ‘That’s cute. What do you do with that? Just give me the files’. He’s like ‘Are you sure?’\nPL: Because my manager, this is true story. My manager told me about it, but it\u0026hellip; So\u0026hellip;\nAS: It was unfair competition.\nPL: He was like ‘Look, if you win this, this is gonna help, you know?’ And I’m like ‘Okay, alright, but also if I win, he wins.’ And so I’m like ‘Alright dude’. And I didn’t do it. On the day of, it was like the week before, he called me and he was like ‘Dude, are you gonna do the video?’ I’m like ‘OK, right\u0026hellip; I don’t know how to edit videos, let me call Adam’. So I call Adam up, he’s like ‘Just put the file up’. So I record the video. He kind of directs me on what to do, he’s like ‘Say the ‘yo’. Do the ‘yo’’. I’m like ‘OK, I’ll do the ‘yo’’. So I do the ‘yo’, I do the video, I don’t say Guy in a Cube, anything like that. And I send it to him. And then I get it back, and I watch it. I was like ‘Holy smokes, alright!’. So I upload it the day after I uploaded. This guy who runs all of worldwide sales\u0026hellip;\nAS: Some vice president.\nPL: I don’t know who this guy is. He sends me an email, he goes ‘Patrick, that video was amazing’.\nAS: He calls me up. He shows me the email and he’s like ‘Someone sent that’. I read it and I’m like ‘No’. That’s a personal email.\nPL: And then I got email after email, after email. Of course I won, right? So we go to the conference, so like a week before, it was in Vegas, a week before they email me, and they’re like ‘You’re a finalist’. With ten people.\nAS: He was worried about it.\nPL: I was.\nAS: He was sweating.\nPL: But when we get there, they’re like ‘Okay’. We won. We were gonna take the top three, so I was in the top three, but we weren’t gonna have you guys present, because we only want the other people to present because then we’re gonna have those top three and then they’ll get an award. But we want all you guys to present to see who’s the winner. Okay, sure. Right now you’re a person. I’m like ‘I’m winning now’. But it really was Guy in a Cube. It really was. And so when we get there, and I’m watching these people present I’m like ‘This is not even gonna be right’. And so the lady goes ‘You don’t present in the first round. You three don’t present in the first round’. I was like ‘Okay’. So then we have the finals. And it’s us three presenting in the final. Four, four. And they all present, and they make me go last. I’m like ‘OK, I’ll go last’. And so I get up there and they have this streaming Power BI report, a created form. And it’s showing the votes, so you vote. So at the end, there was 70-80 people in the room and it was 99% Patrick. And then they bring us out on stage, and there’s confetti, and all this stuff.\nAS: And I hear about it for three days.\nPL: Honestly if it wasn’t for Guy in a Cube, I’m telling you, I am telling you, the managers, the people, especially in sales, it separated me from ‘Hey, here’s the sale, the Technical Sales guys, here’s Patrick’. It really did.\nAS: And all that like in terms of, you know, did it help career, does it help blah, blah, blah. Absolutely. It helps everything. It helped me become a better presenter.\nPL: Yes.\nAS: It helps me in my career, it helps me with communication. It helps with tracking things, like working on things, time management. The thing I’ll tell people is to seriously, like if you’re struggling with getting into something, like if you have a fear whether it’s presenting at your local user group or starting that blog or whatever, just do it. Just put your energy into it. You gotta have\u0026hellip; just start doing it regularly, right? You gotta put some effort into that, and great things will come out of it. I remember when I was in support and I was telling people, I’m like ‘Look, at the time I was blogging once a month’. That’s 12 a year. That’s not a lot, right? It takes a little bit of effort. Just do that little bit and you stand so much above everyone else. Because no one else is doing it. Find the white space, find that area where no one’s doing anything, that opportunity. And just do it. Don’t ask for permission. I didn’t. We didn’t ask for permission on the Guy in a Cube. We just started doing.\nPL: It separates you, man. It does. It separates you. And when you’re working at a company like Microsoft where everybody, not everybody but\u0026hellip; When you’re working with some really talented people, you got to figure out a way to separate yourself.\nAS: But I would even say: just to start doing that one thing that you enjoy and just put yourself into it, don’t worry about someone’s gonna read this. Don’t worry about any of that stuff. Do it for you.\nKamil Nowinski: Don’t worry about that you maybe make a mistake or that kind of fails.\nPL: The video I just did on the two tips, I say ‘Oh, it’s almost half. 21 plus 56, it’s 56... Is 21 half of 56?!.\nAS: I didn’t even catch that.\nPL: And I’m looking at the video, and I’m like ‘Oh my God’. It is way less. You know what I’m saying. You can’t worry about it.\nPK: It doesn’t matter. I watched that Tuesday video, I like it. Being a self-critic is really good in many ways.\nAS: But also with the other thing like even doing YouTube videos. I’ve been doing presenting since 2008, but now even I went back, and listened to some of the recordings of like me in 2010 and 2012, and I’m like ‘Awful. It’s so bad!’. But with the YouTube thing\u0026hellip; The thing I’ve always heard people before, like if you want to become a better presenter, watch yourself present. Record yourself. But I was like ‘that’s just stupid, right? But doing the YouTube thing, this is better than audio, because you’re gonna see your facial. You see your tics and things. Doing the YouTube thing, when I first started, and Patrick got this comment as well, but when I first started, they’re like ‘Dude, your hands are too distracting. You’re using your hands too much’. So the way I fixed that, I told Patrick ‘You wouldn’t do it’. I sat on my hands for like three weeks’ worth of videos, just so I didn’t use my hands at all.\nPL: I have to use my hands.\nAS: That kind of taught me to be a little more like prescriptive with my movements, more intentional.\nPL: If I don’t use my hands, there’s no Patrick.\nAS: I know.\nKN: So what is the hardest? If you are recording, looking at the recorder? Or if you are presenting at the audience?\nPL: Recorder for me. Recorder is really difficult for me. It’s tremendously difficult.\nAS: I will tell you, I love doing the five to ten minutes video. I love doing the full-day pre-con. The one-hour session kills me.\nKN: Why?\nAS: Because it’s longer than five to ten minutes, and it’s not a full day. So I feel restrained in my head, I don’t know what to do.\nPL: I just want to be in front of people. It could be five minutes, it could be six days. I’m OK. I love it.\nKN: So probably we have covered part of this question, but I would like to just repeat it, and make sure that you answer this: how do you find the time and inspiration for your posts or video activities, blogs?\nAS: There are days where I’ll call Patrick or he’ll call me. He’s like ‘We need your video for tomorrow’. I’m like ‘I don’t know what I’m gonna do’. There was one time, originally my videos were dropping on Tuesdays, this was Wednesday. I get up at 5am. 5am I’m like ‘What am I gonna do, dude?’\nKN: Let’s read a newspaper. Then you’ll find some inspiration.\nAS: It’s gotten easier lately just because we actually have more of a backlog, so we know ‘Alright, we’re going to do this’. And actually this year I’m actually really proud of this and I hope we can continue this for the whole year. For January, February and March I’ve actually had it all mapped out, like I know what I’m doing. I’ve already got like half of March already mapped out man like what we’re doing.\nPL: Sometimes like tweets. Sometimes people tweet stuff, and say ‘Hey, how do you do this?’. We get emails. Sometimes you just work on a customer. And you figure out something.\nAS: We got a comment on the video. Originally we got no engagements so there were no comments on the video. But now we get tons.\nPL: Like the video I did with Alberto the other day was purely\u0026hellip; Somebody posted a video and I was like ‘Okay, I’ll write it’. Then I searched it. I searched the Internet and I found this article, I was like ‘Oh, Alberto is gonna be at Bits. We’re gonna do a video on this.\nPK: I remember like sometimes when I watch the video, I go down and actually read the comments. I’m somebody who have been watching your videos from the very beginning like, you know, when you had very few subscribers. And I can totally relate initially there was not much engagement, but now I can see there’s\u0026hellip;\nAS: And I’ll tell someone like whether you’re doing blogging or video, or podcasting, or whatever, you’re gonna go through a struggle of where you feel like ‘Why am I even doing this? What’s the point? Like nobody’s listening to this. I’m just doing this for myself’. And I would say: push through that. Everyone will go through the dip of like the low point where you just feel like it’s just not happening at all. Most people quit at that point. If you push through that, then it starts to take off. And that happened with me. It took like a year and a half, nothing. And I did that YouTube course thing. And I’m like ‘I got to change something. Nothing’s happening’. And, you know, right after that, then the numbers started going up, and since then it’s like everything just compounds exponentially.\nPL: And you can’t take, like you can’t take it personally, right? People are gonna say stuff and, you know, come back and say ‘You could have done it this way. You should do this, you should have done that’. You can’t take any of those comments.\nAS: And you’re gonna get nasty comments.\nPL: You’re gonna get nasty comments.\nAS: I would even say for the women out there listening to this. I talked to the other women youtubers out there, so I’ve talked with Ruth Pozuelo, I’ve talked with other folks that do this, and just other YouTube creators that I’ve interacted with. And I feel like the women, they get bad comments. There are jerks out there. Really. I don’t get those. There was one comment that we got. Patrick didn’t catch it. I called him up and I’m like ‘Dude. That comment\u0026hellip; That was racist.’ Like he called Patrick out on something. Patrick was like ‘I don’t even\u0026hellip; whatever’. But what I loved, and this is something I’ve also heard [\u0026hellip;] stuff. It’s like don’t engage. My audience went in on him. And this is another thing.\nPK: I think know what you’re talking about.\nAS: [\u0026hellip;] got one of those\u0026hellip; There’s two out of the seven or eight. One of them is you’re gonna have your zealots, so these are your hardcore supporters. And then you’re gonna have your haters. Every brand has haters. You’re gonna have them. If you don’t have any haters, you’re not pushing the envelope. And there were some followers like three or four, they just laid in. And I was like\u0026hellip;\nPL: You can’t take it personally though. You can’t\u0026hellip;\nAS: It was fun to read.\nPL: Yeah, but you can’t take it personal. And it’s not gonna be perfect. You can’t take it personal, because everything you record\u0026hellip; You can scrutinize everything. Just like I say it with the video, right? Some might even come back and go ‘Patrick, that was not half’. But okay, whatever. You just keep pushing through it. You keep moving.\nKN: It’s not a Hollywood video, yeah? It’s not Hollywood movie.\nPL: I don’t know. I got a Hollywood director.\nAS: I’m an amateur.\nPL: I don’t know, I’ve got a Hollywood director over here.\nAS: I consider myself an amateur on the video stuff.\nPL: He’s being modest.\nAS: I’m a hobbyist.\nPK: Yeah, I think he’s being modest.\nKN: What hints would you give to young people who wanted to start working in IT market or maybe with Power BI Desktop or whatever?\nPL: You can’t be afraid, you got to start at the bottom. You need experience. Most people, they want the fame, the glory right away. It takes hard work. You got to do the work. You gotta start somewhere. So for me personally, I never went to college. Most people are surprised like, you know, I’m Principal Program Manager at Microsoft, ‘Oh my gosh’. I started doing support for Microsoft on the consumer side, it was a contract position when I was a senior in high school. So I built my first computer when I was 16. It was all experience, right? And it all led up to what I’m doing today. But most people, they don’t want to do the work, they want to jump right ahead to the good stuff. I’m like ‘You gotta do the crappy job. You gotta start there’.\nAS: Work helpdesk. That’s what it takes. When I started my career, I was at mortgage underwriter. I was underwriting loans. You get a house, I would approve the loan, and I took like a half\u0026hellip; my pay got cut in half. Because I wanted to go to IT. And I worked as a\u0026hellip;\nAS: You’re starting all over.\nPL: I was writing query score reports. That’s what I did.\nAS: Do the work. Do the hard work, it’ll pay out. I hear this from folks that are like in their 20s and I’m like ‘Come on, man’. You’ve got so much time in front of you. You’re gonna get there. Just take a few years and do the work. And then, while you’re doing that work, look for opportunities where you can fit in, and do something above and beyond. And you start doing that, then it gets noticed. And then market yourself. Networking is super important. I tell people that at the office, and Bob Ward’s big in the SQL community, and I’m good friends with him at the office, and there’d be people that were like three rows away from where Bob Ward sits. It’s magical, right? Like you’ve got that Bob Ward, Bob Dorr, Keith Elmore who were like the heart and soul of SQL Server support. And you’re right next to them. And I was that way too when I moved to Las Colinas, and I’m like\u0026hellip; I was told, get to know Bob and Bob, and I’m like ‘They’re SQL guys and I’m like on this other team. I don’t have anything’ and then I figured it out, I hacked it. I’m like ‘How do I do this? And I’ll just start participating and asking questions, and engaging, and networking, and I tell people, I’m like ‘Okay, if I go up to Bob right now, does he know who you are?’ The answer’s ‘No’. I’m like ‘Why doesn’t he know who you are?’ Ask him questions, engage with him.\nPL: I think that’s important, right? So opening your mouth. Because I see a lot of people\u0026hellip;\nAS: Be intentional with it though. Don’t just\u0026hellip;\nPL: When we’re at conferences, you can see people pointing and whispering. And I walk up to them and say ‘It’s okay to say hello. I won’t bite you’. If you want to say hello to me, please. I like to talk to people, right?\nAS: We’re like Olaf. We like warm hugs.\nPL: And if I see you, and I watch people. I intentionally, I’ll stay and like I’ll go out here tonight and I’ll watch them, they’ll point. Point, point. And then I go \u0026ldquo;Hey, how are you?’ Got you.\nKN: Yeah, but normally I think… I know that people are afraid to come to you and even say ‘Hello’.\nPL: I like it. I want you to say hello. I want you to give me a big hug.\nAS: At a user group last night, I was walking down the aisle.\nPK: I know. I was very impressed. I was telling you, isn’t it?\nAS: Patrick was like ‘What are you doing?’ ‘I’m talking to the people’.\nPK: I really admire you there going\u0026hellip;\nPL: If you come to my session. And my sessions, the first two rows, I will come out in the first two rows at my session, and I’ll talk to the people in the front row. Because you can see him whispering and stuff. ‘Come on, let’s wave, Patrick, how are you, nice to meet you’.\nAS: I have a question that I ask everyone. It starts the conversation, so I’ll just ask them like ‘If you had a magic wand that could change one thing in Power BI to improve adoption, what would it be?’\nPL: Cause I truly believe that we’re all one step away from where we’re trying to go or what we’re trying to do. And it’s a person. It’s one person, you just got to meet that one person. And if you never walk up to me and say ‘Hello’, how do you know I’m not that person who can say ‘Hey, we are hiring’. ‘Hey, you’re doing this’.\nAS: I’ve always said I’ve looked back, and I always tell people like I’ve been lucky in my career, in my life. I’ve always had mentors, natural mentors. Not like someone’s assigned to me to be a mentor. It was a natural mentor. Bob Ward, he was one of those people. He was probably one the most influential in my life. And I look to him as a father figure. He refers to me as a brother, but I’m like ‘Nah, it’s more than that’. And I’ve said I’ve just been lucky about that. But it’s not luck. I’ve reached out to these people, I engage. I seek out, because I want to learn. I know Bob Ward in the support world, that was the most influential person there. I’m like ‘I need to learn from him’. I just started asking questions. And then he picked up, so I was like, he didn’t know who I was. In fact he actually made fun of me, because he said, when we were doing a keynote, he was telling one of the new folks about how we met and he’s just like ‘Man, who’s this idiot that’s got like this big Star Wars Lego thing on?’ ‘Oh, that’s this Adam guy who just transferred here’. And then I started going to his weekly triage, as asking questions. And then this is how I got into\u0026hellip; Because again, I’m engaging with him. People always ask ‘How do you start speaking?’ Well, for me it was Bob Ward because he recognized, like ‘Man\u0026hellip;’. He came up to things like ‘You ask really good questions’. He’s like ‘You ever thought about speaking?’ I’m like ‘I’ve never’. Audiences don’t bother me. I’ve never done a technical talk before. He’s like ‘Are you okay with it?’ I’m like ‘Yeah, sure’. He’s like ‘Alright, we got this thing called PASS Summit. We have some sessions there. We can get you involved in them’. I’m like ‘Alright, cool’. he’s like ‘Alright, so pick out your topic, go in there’. Alright, great, so a couple weeks go by, and I’m like ‘How long do I need to do? Like an hour, half hour?’ He’s like ‘Well, we like to start people out that are new. We like to start them off a little gentle, less pressure. So we get them started with this thing called a pre-con’.\n[Laughter]\nAS: And I’m like ‘I don’t know what that is’\u0026hellip; And then I’m like ‘Alright, how long is that, 60 minutes?’. ‘It’s like seven or eight hours’. And I’m like ‘What the\u0026hellip; you trolled me on this, didn’t you?’ And he’s like ‘Nah, I didn’t want to do it, and I couldn’t get anyone else to do it’. And so I did the next five years. I was the one doing the pre-con, because it basically now was on me to find someone else, and apparently people were smarter than me. And they said ‘Hell, no, I’m not doing a pre-con’. So I had to do it five years in a row. That’s how I got into speaking. But it’s because I engaged, I asked those questions. And then, as I asked those questions, I did it, then he specifically or someone around there specifically said ‘Okay, we need to move Adam right next to boss’. So I’ve sat next to Bob Ward, Bob Dorr and Keith Elmore for like eight years. And now I’m just learning from them. So what I always say ‘I’m not a SQL guy’. But I’m sitting next to the legends on the support side. Guys that wrote the book on this stuff. And I absorbed it, we went to lunch, we did the walk. We did all sorts of stuff, and so now I’m like I know some SQL stuff, but I’m not an expert on it. But I can hold my own. And this is because of just being around them. But we would engage, and we would do things, and he would coach. I would always say Bob Ward, he’s a smart man, but I always say ‘I go to Bob more for like career and like political type advice. And then Bob Dorr was like the smartest guy that I know. And I would go to him for like technical.\nWe called him the bithead. So I just engaged. I latch on the people that I can get help from, and I can learn from, and my manager one time is like ‘Hey, we need you to get a mentor, you need a mentor. I’m like ‘I got 17 of them. I go to this one for that, I go to this one for that’. And he’s like ‘Okay’. I go to who I need to, I ask the questions, I’ll learn it. So don’t be afraid.\nKN: It’s amazing how much your life can change because of networking basically. OK, so this is the first factor, it’s very important, yeah? So what else? I mean, okay I know all those things are very important, and we covered that, but what else? If you are starting in the IT market or if you would like to start, let’s say, working with Power BI for example? What do you need to do?\nAS: Well, you should watch Guy in a Cube.\nKN: Absolutely, yes.\nPL: And subscribe.\nAS: Selfish plug.\nPL: When I first started in IT, I did everything. I was a developer, a database administrator, I did reporting, helpdesk. Can’t do all that stuff, right? And I meet a lot of\u0026hellip; I call them kids, cause I’m a little older. Kids\u0026hellip;\nAS: You were born in the 1900s.\nPL: I was. And I meet them, and they go ‘I’m a developer, I do some database work, I do graphics, I do this, I do that’.\nAS: Be honest, man. Be humble. Don’t be ass.\nPL: Right. My advice is first, you gotta find one thing and become good at one thing before you think you can\u0026hellip; You gotta focus. So if you want to do Power BI, do Power BI.\nAS: Here’s an example. So like you’re gonna get some entry-level position. Will get in the door, you did some work, you maybe coded something on your own or you figured out how to attach a network cable to a computer, it’s all good. And so like what I did was I started in support. Actually when I started as a contractor, and then I applied for full time employee, I failed that interview twice. And so the third time I got it. And then I’m doing the job, and then I recognized there was this thing called Kerberos that no one wanted to touch. And then it’s weird how I like technology things are like they just come in waves, right? It’s like the week has a theme.\nAnd this week was a theme of Kerberos CRIT sets and CRIT sets were like the server down call that you get from enterprises. And I’ve just kept getting the Kerb calls. And I’m like ‘Oh my gosh, oh this is kind of easy’. But I recognized no one wanted it. I’m like ‘I’ll take it. That’s mine’. And then all of a sudden, within like six months, I had people from Active Directory team coming to me, because I started doing blog posts on it. And now all of a sudden, you type in Kerberos and Adam or Saxton, and you get like a\u0026hellip; The blogs are still reference, and they’re from like 2011 and 2012. I had one of the architects of the Power BI service, we were doing stuff for the Gateway and the Kerberos stuff, they reached out to me, and they’re like ‘Look, you’re like the Kerb guy’. This is from years ago, like a decade ago. I found my thing, right? So you gotta find something and do it. Like I said, find the whitespace, find the opportunity and then make it yours.\nPL: I think in essence a lot of people don’t. They don’t find their thing. They just start to do all this random stuff. Find your thing.\nAS: And then I did it again. So we had reporting services and yet several people doing reporting services. And then they snuck something on people in SQL 2005 Service Pack 2. This whole thing called SharePoint integration came out, and no one knew what the heck that was. Support didn’t even get a heads-up on it. And then of course no one wants to touch that, and I’m like ‘Oh, I’ll do that’. So now I was the SharePoint guy. And I was the Kerberos guy, and then they had the whole claims to Windows token and I’m like ‘Oh, now Kerberos is with SharePoint’. And I made it mine. And then I was the SharePoint guy. So find the whitespace, find the opportunity.\nPL: Yep, find the opportunity.\nAS: Look for those little things that you can do. Don’t do everything.\nKN: That’s also very hard to find out appropriate way, you know. If you are interested in a lot of things, you can’t\u0026hellip; it’s not possible…\nPL: It’s gonna take a little bit. It took me a long time. I was a DBA. I was a good DBA for about seven years, and my manager promoted me and I was managing all these DBAs, I was a horrible manager. The worst manager you’ll ever meet. I made a guy cry. Horrible manager. And I came back from vacation, it was like ‘we need a Data Warehouse’, and none of the DBAs wanted to build a Data Warehouse, they were like ‘I do application, I write queries. I’m gonna optimize them’. I’m like ‘Somebody’s got to do Data Warehouse’, nobody wanted to do it. Two guys actually quit.\nPK: Because?\nPL: They didn’t want to do Data Warehouse.\nKN: From the BI team?\nPL: We didn’t have a BI team. And so we had Oracle SQL Server and MySQL. I’m like ‘Put it on whichever one you want, I don’t care’. ‘I’m not doing it. I’m not doing’. I knew SQL Server, so I was like ‘Okay, I’ll do it’. And I was good at it. I was good at BI. And I was like ‘Well, maybe I shouldn’t be a DBA, maybe I should do BI?’ And I built a Data Warehouse. We had these digital boards. I worked for an advertising company. Every second, or every two seconds, depending on how much the customer paid a spot would play on this board. There was a little web server on it. The devs helped me, they wrote a CSV file and we would capture those CSV files, every 15 minutes we had a DTS package. Yes, I said DTS package - that would run and load our Data Warehouse. And I wrote two Windows services that would pull these directories. And it was loading\u0026hellip;\nAS: DTS for all you young people it’s like the SSIS stuff now.\nPL: You gotta know what DTS is!\nAS: Just helping the crowd here.\nPK: I’ve never worked with the DTS but I know SSIS seems to be the\u0026hellip;\nPL: So it’s now SSIS. And I remember, I got to a point and SSIS came out, we converted it all to SSIS and my Data Warehouse load process would not load every 15 minutes anymore but SSIS was supposed to be so much better. I had no idea why. And I remember this guy, Brian Knight, he owns Pragmatic Works, and he had written some books and some blogs, and I read them. And it was SQL Saturday, and he was presenting at it. I told my boss, I was like ‘You need to pay for me to go to Orlando, because I need to meet these guys. This is gonna help us make this thing fast, cause he’s doing a session on it’. And they didn’t pay. My company did not pay. I said ‘Okay, I’m going home’. I told my wife, I said ‘I need to go to Orlando’. I said ‘I gotta buy me a plane ticket, I gotta stay in this hotel, I gotta go to this event’. She was ‘How much is it gonna cost?’ I said it was gonna cost me about $600, you know. And so she said ‘Okay’. And I had started the user groups, and so I submitted a session. They accepted my session. My session was an awful session.\nIt was using the CLR to perform [\u0026hellip;] in your database. Anyway, I wrote this CLR stored procedures that would capture PerfMon counters and write it. Anyway.\nAS: You should do a video on that!\nPL: But the whole goal was just to meet this guy. And I did another session on doing incremental Data Warehouse loads with the stuff that I had written. And the guy, the president of the company was sitting in the back room. I didn’t know and I present it, and so then I went to Brian’s session, and I waited, and I waited, and I told him about\u0026hellip; he was like ‘Are you using the Slowly Changing Dimension transform in SSIS?’ I was like ‘Yes’. He was like ‘Don’t use it!’. I was like ‘Is that it?’ He goes like ‘Guaranteed, that’s what you did’. I told him about the two dimensions that were slow. Sure enough. That was the answer, right? But before I left, the guy, president of the company had seen my session. And Brian said ‘Are you the one that did the incremental blow thing?’ I was like ‘Yeah’. He was like ‘I want you to go talk to this guy named Adam’. There’s another Adam. Adam Jorgensen. ‘About working for us’. I was like ‘Really?’. And I went, I talked to Adam, and he was like ‘What would it take?’ I was like ‘Make me an offer’. I was living in Baton Rouge, Louisiana. When I got off the plane, my cell phone was ringing. It was Adam making me an offer. And I was so happy to take that job, because I wanted to go back and tell them ‘If you had paid for me to go to Orlando, I wouldn’t have taken the job’.\nKN: But seriously, we are talking about the choices in your life.\nPL: You have to invest in your life.\nPK: I totally agree on that.\nPL: I made the investment. Because it was slow and I didn’t know why.\nPK: The very first SQLBits I attended, my company didn’t want to pay, so I paid myself, but SQLBits changed my life. I would have not been in the position where I am.\n**KN: Like I started from the local group and many else from events like this, but in Poland.\nAbsolutely, it will be two episodes from that recorded. Can we cover two more questions? How many people or teams are working on Power BI Desktops or services at all? In total.**\nAS: I don’t know what the total number is. There’s a bunch of people working\u0026hellip; From developers to PMs, to like\u0026hellip; Even Power BI Desktop has multiple PMs on it. I think there’s like five or six PMs on it now. Service team has another seven or eight\u0026hellip;\nPL: We have engineering\u0026hellip;\nAS: There are engineering managers, the devs\u0026hellip; The devs are really like the unsung heroes.\nPL: Nobody knows those guys.\nAS: I know these guys\u0026hellip;\nPL: Yeah, but out here.\nAS: They’re fun guys. I actually like hanging out with the dev guys, cause they’re the ones that actually know what’s going on. Because they’re doing the work.\nPL: And then we have interns that come in, do stuff.\nAS: Summer interns.\nPL: There’s a lot of people working on it.\nAS: You’ve got the marketing teams, you’ve got\u0026hellip;\nPL: You’ve got us.\nAS: You’ve got us. You’ve got the support teams. There’s a lot of people. It’s a lot of work.\nKN: No wonder that every single month you have a big update.\nAS: I mean, it’s a huge team that cranks out a lot of features. Honestly, it’s amazing that the amount of features that they do\u0026hellip; I don’t think they have enough numbers. They’re cranking out more than what you think they should be doing, right? It’s just amazing to watch them.\nPL: It’s pretty impressive.\nAS: And the way that they can turn around on something based on customer feedback or issues. I remember there was one issue where the filter\u0026hellip; was it the filter or the Drill Down\u0026hellip; one of those features where it was on the visual, I think it was the Drill Down. And we moved it up into like the bar, the menu bar up top. And I got a support call on it like ‘I don’t know where this is’. I’m like ‘I’ll do a blog post on it’. So it was like ‘Hey, where did my Drill Down go?’ And oh my gosh. That blog post\u0026hellip; There was so much like\u0026hellip; Normally I get like two or three comments on a blog. This was on the Power BI blog and it was something like 700 comments. It was everyone that was pissed. It was like visceral comments on it. And I’m like ‘Oh my god, what did I do?’ I was just trying to help you to tell you where it went and they’re like ‘Why did you move it?!’, blah, blah. In less than 24 hours, it got moved back because of that. I remember, the group engineer, the person that was there before, Kim Manis.\nCause Kim Manis [\u0026hellip;] the Desktop now was the guy before. And she paid me, and she just like ‘Wait, you’re in support, right?’ I’m like ‘Yeah’. And she’s like ‘You post it on the Power BI blog? And I’m like ‘Yeah, I may do that’. And I talked to her, she’s just like ‘Yeah, we’re gonna work on it. I’m moving that back because of the blog. And I’m like ‘Alright, you need to comment on that because that’s not me’.\nKN: Apparently the users have a power to change software.\nPL: Absolutely.\nAS: Well, there’s an example. I think it was David Eldersveld who said something about ‘Hey, can we just get like custom tool tips on\u0026hellip;’ I forget what it was, images or something, on the buttons. People just retweeted it and Amir caught it, right? And then, within a couple weeks, it was in the product. So people can effect changes. This is why I actually, I say this, it becomes a joke, but I’m very serious. Vote things up on ideas at powerbi.com. Seriously. Strengthen numbers.\nKN: In terms of Power BI and very popular topic like DevOps. What do you think, when will the team bring something that will enable automatic deployment?\nAS: I mean, we already have things today for that. We’ve got APIs for cloning reports or moving things from workspace to workspace. I will admit there’s some gaps in that, but there are some approaches that you can do. There’s things that are being worked on that are in the business application release notes in terms of like [\u0026hellip;] type functionality. We’ve got the XMLA endpoints that are coming out which will also help with those type of operations.\nKN: Yeah, we heard that, it’s coming, yeah?\nAS: So the Read part of it is coming soon, and then the Write axis for that will come later on. It’s not in the current semester. And they do planning in six months semesters. Just keep your eye on the release notes in terms of when, it’ll be listed there once it’s coming. But I mean, there’s work that’s being done right now to help with those items, and so they’re working on small things to help the new workspaces. I think we’re going to enable a lot of functionality in that front, and make it a little easier. Because today we’re tied to the Office 365 Modern Groups. It actually hinders us from certain things, whereas once the new workspaces are there, we have a little more flexibility to move things easily, and to link things. Things like shared datasets or you can have a dataset of reports in different workspaces, and easily copy things from one thing to another. Just there’s a lot of stuff being worked on. It will get better.\nKN: We’ll be following this.\nPK: Yeah, I’m so looking forward to new workspaces coming out of the preview with all the features and stuff.\nKN: Okay, guys. Thank you very much for this!\nPK: Thank you so much.\nPL: You’re welcome!\nUseful links Guy in a Cube: WebSite | YouTube | Twitter | Facebook | Instagram | Snapchat\nAdam Saxton - Twitter\nPatrick LeBlanc - Twitter\nProducts: Microsoft Power BI | Ideas\nEvents: SQLBits Website \u0026amp; Twitter | Techorama | Power BI World Tour\nCommunity: London Power BI User Group | Meetup\n","date":"2019-05-31T06:00:40Z","image":"/uploads/2019/05/ASF022-SQLBits-with-GuyInACube_1200x700.jpg","permalink":"/2019/05/asf-022-guy-in-a-cube-interview-part-2/","title":"ASF 022: Guy in a Cube interview (part 2)"},{"content":"It was tremendously extensive week for me (a week ago) when I spent time between two conferences: SQLDay (Wroclaw, Poland) and SQL Saturday #857 (Kiev, Ukraine). I\u0026rsquo;ve been helping and speaking. Between the events, I had to sort out certain private stuff. You right - that\u0026rsquo;s the reason why I omitted previous \u0026ldquo;Last Week Reading\u0026rdquo;.\nPress ADF Mapping Data Flows: Optimize for Azure SQL Data Warehouse\nWhat\u0026rsquo;s worth to consider when Azure SQL DW is your sink (target).\nADF Mapping Data Flows: Create rules to modify column names\nMark Kromer presents a pattern of chaning column name based on their data type.\nComparing Mapping and Wrangling Data Flows in Azure Data Factory\nCathrine nicely explains the differences between these two. You might not have heard about Wrangling Data Flows yet also.\nPower BI Desktop May 2019 Feature Summary\nAmanda Cofsky\u0026rsquo;s about the freshest features in Power BI.\nPower BI open-platform connectivity with XMLA endpoints public preview\nXMLA is like a Connection String to Power BI Workspace/Dataset in Premium Tier.\nDAX 101: Automatic time intelligence in Power BI\nThis article shows why building custom Date tables is preferable to using the automatic date/time handling capabilities of Power BI.\n12 Version Control tools for SQL Server\nThere are more tools than only SSDT and Redgate thing\u0026hellip;\nGuyInACube reached 50k subscribers. Adam confesses himself with the contents of his backpack.\nFlattening JSON in Azure Data Factory\nIncluding more complex JSON structures\nA word from Microsoft’s CEO An important piece by @XboxP3 on the opportunity in gaming – and the responsibility that comes along with it. https://t.co/Abhrzwtcfd\n— Satya Nadella (@satyanadella) 20 May 2019 Video: These self-driving tractors could make farming easier and greener https://www.youtube.com/watch?v=QSnMDjYMe6U\nSmile Corner ","date":"2019-05-26T16:15:41Z","image":"/uploads/2019/05/news-absolutvision-365898-unsplash-390.jpg","permalink":"/2019/05/last-weeks-reading-2019-05-26/","title":"Last Weeks Reading (2019-05-26)"},{"content":"Hello folks! One week passed, another one almost began. And that will be a crazy week on two conferences. Today, I\u0026rsquo;m in Wroclaw (Poland) where we\u0026rsquo;re prepping SQLDay conference - the biggest conference about Microsoft Data Platform services in this part of Europe. On Friday, I\u0026rsquo;m going to Kiev for SQL Saturday in Ukraine. Busy week!\nPress Get the YTD of the same period last year\nKasper (T) is doing reexamining of his one of the most (10-years old btw) popular blog post, still valid.\nThe Most Valuable Company (for Now) Is Having a Nadellaissance\nHow that does happen that Microsoft has overtaken Apple to become the world’s most valuable company.\nMicrosoft Build 2019 Event Roundup\nHere Are The Biggest Announcements.\nAnalytics in Azure remains unmatched with new innovations\nEasily transform data at massive cloud scale in a code-free design environment - Full public preview now!\nSQL Database serverless (preview)\nPause and resume a single database and base on billing per second.\nRoadmap for Enterprise Modeling with Power BI and Azure AS\nChristian Wade (T) discusses the future of Enterprise BI solutions on the Microsoft Platform.\nShould I pluralize table names, is it Person, Persons, People or People?\nHard to do not agree. Check the arguments if you have such discussions in your team/company.\nA word from Microsoft’s CEO Fantastic to have @melindagates back on the Microsoft campus today to talk about her new book, #MomentofLift. It was a powerful and timely conversation about the importance of lifting up women to lift up all of humanity. It was inspiring to hear her stories and insights. pic.twitter.com/qsDzyAwNC8\n— Satya Nadella (@satyanadella) 10 May 2019 Video: Coca-Cola on Azure Cosmos DB https://www.youtube.com/watch?v=DqQuXGjaE8c\nSmile Corner ","date":"2019-05-13T09:34:26Z","image":"/uploads/2019/05/news-absolutvision-365898-unsplash-390.jpg","permalink":"/2019/05/last-week-reading-2019-05-12/","title":"Last Week Reading (2019-05-12)"},{"content":"Azure Data Factory is more of an orchestration tool than a data movement tool, yes. It\u0026rsquo;s like using SSIS, with control flows only. Once they add Mapping Data Flows to ADF(v2), you will be able to do native transformations as well, making it more like SSIS.\nA couple of days ago (May 6th, 2019), Microsoft announced Azure Data Factory - Mapping Data Flow became to Public Preview. For many months I’ve had the pleasure of Private Preview access to test many scenarios, see how the product has evolved and how quickly review new features as they are added. I\u0026rsquo;m glad that I add my small contribution by raising some issues and helping to make it better. From today, you also have the chance to check how these functions work.\nMapping Data Flow In the early Private Preview stage it was called Data Flow (without Mapping). However, if you know more tools and services from Microsoft\u0026rsquo;s stage - you knew that Power BI has had Data Flow too. And that was confusing because mainly they are a different set of features, except the fact both work on data.\nAt first, MDF (Mapping Data Flow) seems to be just another component on the pipeline map. However, if you look closely, you will notice that this component can do a lot more. A single MDF can contain a whole series of many transformations to finally obtain the desired data format, their granularity, quality and get rid of unwanted rows. If you know SSIS - it\u0026rsquo;s like Data Flow in SSIS (not exactly the same - I will come back to this topic later).\nMapping Data Flow must contain at least one source activity. The simplest version of it may be like this:\nJust pull and push the data But assuming you are copying data with like-for-like schema, it does not sound sensible because the same action can be made in ADF\u0026rsquo;s pipeline using \u0026ldquo;Copy Data\u0026rdquo; activity.\nThe situation will change very quickly when we add the desire to have an additional column. Standard \u0026ldquo;Copy Data\u0026rdquo; does not meet such a requirement. Mapping Data Flow comes with help where we can design a lot more complex data processing processes.\nSeveral more components don\u0026rsquo;t make the Data Flow unreadable Available activities Let\u0026rsquo;s review what kind of options have been brought to the table.\nAt the moment you have the following operations (activities) available in Mapping Data Flow:\nSource New Branch Join Conditional Split Union Lookup Derived Column Surrogate Key Pivot Unpivot Window Exists Select Filter Sort Alter Row Sink I\u0026rsquo;m sure that more activities appear in the future.\nNow, having Mapping Data Flows you\u0026rsquo;re able to build Slowly Changing Dimension loading processes in ADF. If you want to see an example - create a pipeline from a template.\nAzure Data Factory - Pipeline template gallery Getting Started with ADF Mapping Data Flows I\u0026rsquo;m not going to show you how to build the Mapping Data Flow step by step using a variety of activities. I will do a series of posts about selected activities of Mapping Data Flow to present them in full detail. In the meantime, the better way of introducing you to Mapping Data Flow would be video presentation I have given a few recent conferences. The last time I presented this topic was at the SQLBits conference in Manchester, UK.\nHere you are - the entire video about Mapping Data Flow only:\nVideo: Mapping Data Flow - capabilities (SQLBits 2019 conference) Is Data Factory SSIS in the cloud? Generally yes, but\u0026hellip; not thoroughly. There are a few aspects that need to be taken into account.\nFirst of all - YES, because now we have capabilities to cover business logic with the data, such as data cleansing, transformation, aggregation, calculation. Therefore, you might perform all data preparation before pushing them to Azure SQL DW or ADLS (Azure Data Lake Storage).\nAnd NO, because you can\u0026rsquo;t simply migrate your existing SSIS packages to Azure Data Factory. Although many similar operations exist, not everything is possible. For example, Event Handlers are not present as we know it from SSIS, as well as 3rd party components we could install in on-prem SQL Server Integration Services (SSIS).\nSSIS ADF Feature Control Flow Pipeline Feature Data Flow Mapping Data Flow Connections Connection Managers Linked Services 3rd party Components Templates If you are interested in taking a deeper look - check this page out where I compared SSIS components available in Data Flow to their equivalents in Mapping Data Flow.\nTurning on Data Flow Debug mode As you can see, there are many similarities between these tools. What’s happening behind the scenes is slightly different, though.\nAzure Data Factory is metadata only. I mean, it doesn\u0026rsquo;t have its own engine. How does it go then?\nWhen you run a pipeline which contains Mapping Data Flow component (activity) or run Debug mode in MDF - a Databricks cluster is spun up underneath for you. The entire Data Flow will be converted to Scala, compiled and built as JAR library and sent to the Spark Cluster.\nAzure Databricks is an Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform. The Databricks Runtime is built on top of Apache Spark and is natively built for the Azure cloud. Azure Databricks integrates deeply with Azure databases and stores: SQL Data Warehouse, Cosmos DB, Data Lake Store, and Blob Storage.\nThe key feature of Azure Databricks is its ability to blend and transform data at scale. That\u0026rsquo;s why Azure Data Factory and Databricks are ready to go with Big Data processes and plays a significant role in many modern Business Intelligence and Machine Learning solutions.\nAn Azure Databricks cluster utilised by ADF is fully managed by Microsoft and you have not got access to it.\nData exploration Data Preview panel in ADF Mapping Data Flow One of the greater features in Mapping Data Flow is Data Flow Debug mode and capability called Data Preview. At each stage of the Mapping Data Flow (MDF) process, selecting any component - you can explore outgoing data in the bottom panel, which presents all columns, their data types, as well as basic statistics for the selected column.\nThat panel also shows you how many rows have been added, updated, deleted or upserted at this stage.\nWhile selecting a column - on the right panel you\u0026rsquo;ll see statistics and metrics like values minimum, maximum, std deviation, percentiles (25, 50, 75), variance and average. Obviously, not all will be available for non-numeric data types.\nPretty cool, isn\u0026rsquo;t it?\nData Flow Execution Plan Execution Plan shows you the whole road that data goes through from the source up to the sink (target) during an execution. Furthermore, the diagram is very readable and contains a lot of useful audit information about the executed flow. The general view presents all the sinks and transformations which took part in the Data Flow and information such as the number of processed rows, execution time per activity or whether a table has been cached by Spark or not.\nMapping Data Flow - General View When you click on a sink - you\u0026rsquo;ll see:\n1) right-hand side panel - information about columns of that sink (target), stream information and partition chart.\n2) bottom side panel - mapping for all sink\u0026rsquo;s columns and their origin (sources), including applied transformation methods.\nSource \u0026amp; Target (sink) Azure Data Factory is an orchestrator for data pipelines and supports many (90+) various connectors, including structured, semi or unstructured sources. As a mature tool, it supports multiple scenarios of working with files or database tables, either on source side or destination.\nMultiple useful capabilities are supported for connectors, such as:\nSource transformation Sink transformation - wildcards, file sets, - move file / Delete file, - auto-detect types, - schema validation, - query statement - output to a single file, - clear folder, - truncate table, - recreate table, - naming patterns Also, both, source and sink transformations support CSV/TSV and Parquet file format.\nAlternative As I mentioned above, Mapping Data Flow is converted to Scala to run it on Databricks. If you are more like a developer and love writing code - you can write the same process using all capabilities of Azure Databricks. Write down the code in Python, Scala, SQL, or even mixture of them all in one/multiple Databricks notebook(s), then execute them directly from Azure Data Factory leveraging Notebook activity.\nDatabricks Notebook Activities in ADF pipeline It\u0026rsquo;s a very reasonable alternative in a case when you have more Python related skills in your team or do migrate existing Scala or Python solutions.\nHave a look at this post which compares how to do the same things in Azure Databricks and with ADF Mapping Data Flow.\nConclusion Azure Data Factory is one of the important components in every architecture diagram when building a modern data warehouse solution. Mapping Data flow has been a missing piece in the Azure Data Factory orchestration tool. Now, having that user-friendly UI which allows you to build end-to-end Big Data processes without the need to write code, means not only developers might use the service, but also teams of Business Analysts as well as Data Scientists.\nIt can successfully replace SSIS, especially in many greenfield projects.\n","date":"2019-05-08T07:15:01Z","image":"/uploads/2019/05/azure-data-factory-mapping-data-flow.jpg","permalink":"/2019/05/mapping-data-flows-in-adf-v2/","title":"Mapping Data Flow in Azure Data Factory (v2)"},{"content":"Hello folks! Another set of information about the Microsoft Data Platform. This time: Spark, SQL Edge(!), Power BI, Big Data, Pester, DevOps, YAML.\nPress Microsoft brings Azure SQL Database to the edge (and Arm)\n\u0026ldquo;First SQL Server on Linux, then SQL Server 2019 Big Data Clusters, and now Azure SQL Database Edge; all with one integrated experience!\u0026rdquo; - Slava Oks.\nPower BI Mobile Reporting\nIf you interesting how to design a good mobile report you can find some good tips in that post.\nDeploying Blob Containers with ARM Templates\nWhen you interested in what to add to an ARM Template in order to deploy Blob Container.\nImproving Performance In Spark Using Partitions\nNo shuffle means better performance, not only with Parquet files.\nSpeaker Tips \u0026amp; Legit Feels\n10 important tips for (new) conference speakers.\nUpdated Analytics and Big Data comparison: AWS vs Azure\nThomas LaRock made a brief comparison between these services.\nIntroduction to Testing Your PowerShell Code with Pester\nThe first post in a series about Pester.\nBack to Basics: Power BI Relationship Demystified\nSometimes it’s good to back to roots. In this post, you can refresh your knowledge about Power BI Relationship.\nAzure DevOps Roadmap update for 2019 Q2\nIs it weird if I see YAML everywhere?\nA word from Microsoft’s CEO We're partnering with the U.S. Department of Veterans Affairs to bring the Xbox Adaptive Controller to veterans with limited mobility, connecting them to the games they love and the people they want to play with. https://t.co/XPcashzQTU\n— Satya Nadella (@satyanadella) 30 April 2019 Video: How 5G will change your smartphone, and your life in 2019 https://www.youtube.com/watch?v=MC_Sfkh5-zQ\nSmile Corner ","date":"2019-05-05T23:24:01Z","image":"/uploads/2019/05/news-absolutvision-365898-unsplash-390.jpg","permalink":"/2019/05/last-week-reading-2019-05-05/","title":"Last Week Reading (2019-05-05)"},{"content":"An unusual week has passed. SQL Server\u0026rsquo;s 30th birthday - learn a bit about history. Future? Here you are! .NET enters under Spark\u0026rsquo;s thatch, and more\u0026hellip;\nPress Happy 30th Birthday Microsoft SQL Server\nHave you seen the Bill Gates and Edward M. Esbar Jr. video promoting SQL Server 1.0?\n.NET for Apache Spark\nYes, you have heard very well, \u0026hellip;\nIntroducing .NET for Apache® Spark™ Preview\n\u0026hellip; and here you have more details.\nRemoving multiple patterns from a string\nHugo Kornelis (T) explains step by step how he has built it in pure T-SQL.\nGrouped Aggregate Pushdown\nQuite deep and interesting article about factors that affect aggregate pushdown for GROUP BY queries only.\nASF 021: Guy in a Cube interview (part 1)\nYoooo! The new episode went off. This time, along with Prathy, we\u0026rsquo;re interviewing Adam and Patrick.\nA word from Microsoft’s CEO Congratulations Melinda, on the launch of your book. It's such a valuable contribution to a topic that couldn't be more important to society today. I've always learned so much from the stories you've shared – and am encouraged that more people will benefit from your insights. https://t.co/Hs1nO63MMr\n— Satya Nadella (@satyanadella) 23 April 2019 Video: Code-free modern data warehouse\u0026hellip; https://www.youtube.com/watch?v=E0ZLHwVbAIA\nSmile Corner ","date":"2019-04-29T22:59:46Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/04/last-week-reading-2019-04-28/","title":"Last Week Reading (2019-04-28)"},{"content":" Introduction Guy in a Cube is all about helping you master business analytics on the Microsoft Business analytics stack to allow you to drive business growth.\nThey are just two guys that do the work.\nAdam \u0026amp; Patrick look at how to leverage Microsoft Business Analytics to allow you to gain knowledge that is needed to shape the data your business cares about. This includes Power BI, Reporting Services, Analysis Services and Excel. If you work with our business analytics products or services, be sure to subscribe and join in the discussion with their weekly content on YouTube channel.\nAdam Saxton and Patrick LeBlanc are Microsoft employees.\nThis talk has taken place during SQLBits 2019 in Manchester (UK) on 28th February 2019 (Thursday).\nInterviewers: Prathy Kamasani (T), Kamil Nowinski (T).\nDo you want to know why actually YouTube?\nWhat tool Adam does use for post-production?\nWhy the very first movies were like \u0026ldquo;awful\u0026rdquo;?\nWho has been scared to death and why?\nWhat were the circumstances of joining Patrick to the team and what\u0026rsquo;s fun fact related to his name?\nAnd finally: how did the YOOO become a hallmark - get know the entire story about it.\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript Kamil Nowinski: Thank you guys for accepting the invitation for this podcast Ask SQL Family. So the first question is very hard. What is your name and where do you live?\nPatrick LeBlanc: Patrick LeBlanc and I live in Atlanta, Georgia.\nAdam Saxton: Ask him what LeBlanc means.\nPL: The white.\n[laughter]\nPL: En français, en français. It means ‘the white’.\nKN: What do you both do for a living?\nAS: I’m Adam Saxton by the way.\nPL: So, Patrick LaBlanc, I live in Atlanta, Georgia and I’m a Program Manager on the Power BI CAT team. What’s your name?\nAS: I’m Adam.\nPL: What’s your last name?\nKN: Full name.\nAS: Awesome Sauce.\nPL: Awesome Sauce.\nAS: Saxton. And I live in Houston, Texas.\nPL: Go Texas.\nAS: And I’m a PM on the Power BI CAT team.\nPrathy Kamasani: And he has an awesome house\u0026hellip;\nAS: We have an awesome house. And I’ve got an awesome studio in my house for the video stuff.\nKN: I’ve seen that there will be one question about it.\nAS: It’s bigger than some people’s houses, so\u0026hellip;\nPL: I like it, I’ve been there. I like it.\nPK: I think next time in Texas\u0026hellip;\nAS: Swing by, we’ll do a video.\nPL: I like it.\nKN: Everything is so big in Texas?\nAS: Everything is bigger in Texas.\nKN: Really? Bigger than other parts of the USA?\nPL: (Whispering) No it’s not.\nAS: It is.\nPL: I grew up in Texas, no it’s not.\nAS: He was born in Houston.\nPL: I was born in Houston, Texas.\nAS: His whole family is there.\nPL: Everyone but me. I have. 7 of us. And I’m the only kid that doesn\u0026rsquo;t live there.\nAS: He says he’s coming over to Houston to visit me, but really, he’s visiting his family. And I just happened to be there.\nPK: I visit Houston at least once or twice a year.\nPL: We talked about this.\nPK: Most of my family are there.\nKN: So you both are working for Microsoft, yeah?\nAS: Yes.\nKN: So what is your role basically? What are you doing on a daily basis?\nPL: I like the way you explain it.\nAS: So our team, the Power BI CAT team, because everyone says: ‘What’s the CAT team?’ I’m like, we are a team of elite ninjas that drop in and get shit done.\nKN: Get shit done, nice explanation.\nAS: So we help customer deployments, pre-sales, escalations, so if someone escalates to like Satya Nadella (Microsoft CEO) or Scott Guthrie and it’s Power BI-related, it comes to our team. We get it done.\nKN: Nice.\nPL: We fix stuff.\nAS: We basically get to play with all the cool stuff, do road map discussions, we present, we travel the world, we get to\u0026hellip; We have inputs into the engineering team to effect change in the product based on customer feedback. It’s very exciting.\nKN: So if you have a very good idea, we go to you, yeah?\nAS: No, you go to ideas.powerbi.com.\nKN: But if it’s a really good idea\u0026hellip;\nPK: Yeah, if it’s a very good idea, you can obviously go ideas.powerbi.com. And I think it’s more likely if you are having an issue\u0026hellip;\nAS: That’s when you call support. So if you are a big customer and you’re trying to get something done and deployed, and work on some cool projects and that’s usually when we get involved. We deal with enterprise, big customer\u0026hellip; I don’t know, name a customer.\nPL: Name someone who’s big\u0026hellip;\nPK: The client where I am working? My client is a big customer for\u0026hellip;\nAS: Can you say the name?\nPK: BP. British Petroleum.\nPL: Yeah, It’s a big customer.\nAS: We’ve got show cases. Like at the Gartner Conference last year. So, Adobe and Rockwell where they did customer session on Power BI, of their deployments at the Gartner Conference, so those are some of our customers as well.\nPK: Basically you’re in the cool team.\nAS: The struggle with our team also, and I had this issue when I was in support for 10 years as well. And it was the same thing where nobody calls support or calls our team when they’re like ‘Man, Power BI is just awesome’. That doesn’t happen. They call us when something bad is going on. We see all the bad stuff. We get it fixed and help.\nKN: Exactly, but that means you can improve something. You know what you need to improve.\nAS: But it also means we get depressed. I’m like ‘Oh my gosh, Power BI is just not\u0026hellip;’ and then we go back with the community, we do stuff at conferences that brings our hopes back up, because we know that there are people out there that love it and are doing awesome things. Our customers are doing awesome things with it too. It’s just they call us when they’re hitting a struggle of some kind. And we’ve got take care of it.\nPK: Actually I wanted to ask this question later, but we are talking about the team, that’s why I want to ask. So when you are in a CAT team, you deal with pretty much everything related to Power BI, right? So how do you keep up with all that happening?\nAS: He watches my round-up videos.\nPL: I actually just joined the team back in November. And I was like ‘Holy smokes, this is gonna be a lot of stuff’. But there’re guys on the team that are really good at specific things. So we have one guy, like embedding and security, so you know a lot about these things, when it starts to get into the weeds\u0026hellip;\nAS: We’re a team.\nPL: You just say ‘Hey, Adam, I’m trying to figure this out, I need your help’. So it would be great to know 500 level on everything. You just can’t, right? So there’s a data flow guys, a security guy, embedding guy. I can but I’m not a normal person, right? Like a superhuman.\nAS: I was doing a memory dump yesterday on Power BI desktop. And then I ran an XPERF to find the memory allocation error, it was good stuff.\nPK: I know, when you guys did that, when you showed us at the London Power BI Meetup, I was like ‘wow!’.\nAS: Truthfully, so the round-up videos I do, those actually really help me to stay up to date, because it forces me to read the blogs. Especially the update blogs that we put out and then like what other folks are doing in the community. And there was some time where I had to miss, I think I missed like two or three round-up videos, and I felt lost, I’m like I don’t know what’s going on with Power BI now. So it’s like the round-up videos actually really helped me to stay up to date.\nPL: And believe me or not, the stuff that you guys do, we read that stuff. And I learned, I do learn a lot from the stuff that people outside of Microsoft do. I learned a lot of stuff internally, but the blog posts, videos and all sorts of stuff\u0026hellip; We read that stuff, we watch that stuff. A lot of my videos come from that. I mean, I modify a little bit, and I shout people out, I’m like ‘Oh, I was working with such and such’.\nAS: We reach out to folks too.\nPK: I personally, like whenever I get tagged, that’s like\u0026hellip; People can say ‘what’s happening?’. I was just like ‘I’m just happy, because I was tagged!’.\nKN: Yeah, that kind of things happen.\nPK: I’m just happy. People can just see it on my face. Actually, I wanted to ask a question\u0026hellip;\nAS: So you say you’re happy, and they’re like ‘Why are you so happy? Guy in a Cube? What?!’ It’s a whole different discussion.\nPL: It’s two of them by the way! And they’re both in a cube together. What are doing in this cube, I don’t know.\n[laughter]\nPL: You got the wrong person on the microphone.\nKN: So guys, are you working remotely, from your houses or how?\nPL: I work in a cube.\nAS: Seriously, actually when I moved down to Houston, I actually thought about buying a cubicle for my office, because of the whole Guy in a Cube thing. Because I was actually in a cube at the Las Colinas office in Dallas/Fort Worth, that’s where it all started. But I’m like ‘No, get done with the cube’. It’s a cube in spirit, but yeah, we work remotely.\nPL: Both work remotely. I work in a basement.\nAS: I work upstairs.\n[laughter]\nAS: We both have couches there, and we sometimes have to sleep on.\nPL: Yeah, I have a couch, I have my desk and my whiteboard and everything. What you see in the video, that’s where I work. That’s exactly where I work.\nAS: Yeah, where I’m at is my office. I have a long commute, I have to go up like a flight of stairs. He’s gonna go down a flight of stairs.\nKN: Both of you, do you have appropriate silence in your studio?\nPL: Mostly.\nAS: Mine is pretty quiet just cause I’m out in an area that’s like out in the middle of nowhere. So the neighbourhood is very quiet where I live now. When I was in a rental home before, we just moved into the house I’m in about a year ago. So before that I was in a rental home in the Houston area while we were looking for a house. And there happened to be a major road right behind the house. That was painful. It was like every 5 seconds, I had to stop. So horrible. So I started getting up at like 4 am, cause there was no traffic.\nKN: That’s why you’re so sleepy in some videos.\nAS: If you could see the outtakes on those videos, every other word was like\u0026hellip; I was just\u0026hellip; Oh.\nKN: But seriously, I’m thinking about your daughters for example.\nAS: Yeah, they’re at school. I will tell them I’m about to record videos, and they get all pissy at me, because now they know they have to go to their room and be quiet. The other problem I have though is where my house is, so like the main family room area and then the stairs that go up to my office, and it’s like a megaphone. So anything that happens in that family room is just blasted upstairs. So I’m like ‘you guys have to disperse’. So that’s why I like when school’s in session, because they’re not at home. Also on Saturdays, my daughter has horse riding lessons. My wife will take her to horseback riding lessons, it gives me good 2-3 hours, and she does that, so I can record the videos while they’re gone. My older daughter just stays in her room anyway.\nPL: I don’t have these problems. Because in Georgia, unlike Texas and Louisiana, you’ve got basements. Until my basement is finished, it’s two other storeys above it. My kids are floor above me. So the middle floor is where my wife is, but she’s never there. And the kids are way up, so nobody ever comes there.\nAS: Unless the lawn guy comes.\nPL: Unless the lawn guy comes. But besides that, I just close the door and nobody comes down. I’m just down there by myself.\nAS: Nobody likes Patrick.\nPL: Nobody likes me.\nPK: Your children are\u0026hellip;\nPL: 18 and 14. One’s graduating this year.\nAS: Lucky man.\nPL: Yes, one more. 4 more years.\nKN: So when we’re talking about the videos, how long does it take you to record 15 minutes of?\nPK: Actually before that I want to ask a question. Why YouTube?\nAS: Why not?\nPK: Well, of course\u0026hellip;\nKN: They have any alternatives?\nPK: No, no. Why did you go for\u0026hellip; Why did you think of recording and putting it there?\nAS: As opposed to what…?\nPK: …like blogging? Generally, I love the concept of YouTube. I totally love it. But I just want to understand why did you start.\nAS: I’ll answer the question with the question, which you should never do, but it will lead into the discussion. If you have a problem and you’re trying to figure out how to do something, where do you go to ask the question?\nPK: Forums…?\nAS: Most people go to YouTube. It’s the number 2 search engine in the world, next to Google. And also Google’s gonna surface all the YouTube videos also, so that’s where the audience is.\nPK: OK.\nAS: That’s why I went to YouTube. I started doing blogging way before I did the video stuff. Originally, when I started doing the video, I was embedding the videos in the blog, so I had the blog with the video. Then I just dropped the blog and I only do the video. But I did look originally at doing like Channel 9. Again, it was an audience thing. The numbers aren’t there as compared to YouTube. So YouTube’s got the platform and that’s where I went.\nPK: I was more like thinking, because you are quite geeky about the older stuff. So is it that’s why you went for the videos? Because you already liked photography, or is it because you saw the platform?\nAS: I knew nothing about video or photography, I didn’t know anything about exposure, aperture, ISO levels, none of that before I started doing all of this. I just learned.\nKN: But you got some course, I remember our discussion last year.\nAS: First upload was December 2014 and then February 2016, so it was more than a year later, I only had like 400 subscribers and all the number were flat. I get no comments, no engagement, no nothing. So I was like ‘Alright, we’ve got to fix this. We’ve got to do something’. There was a YouTube certified guy that I followed, and he did a course where he brought in 20 creators. You had to pay for it, but they brought us all in and every week we met on Zoom and whatnot and we had a discussion with the guy, we talked about different topics, and then we had homework and I had to review a ton of channels, to the point where my recommended videos on YouTube were like make-up videos, which, I will tell you, if you wanna learn about lighting, go watch make-up videos. And then also usually the make-up YouTube personalities, they will have behind the scenes, like how they record stuff, so you can actually see their setups. What lights kits they use and all that. So it was actually really helpful to watch this. I watched a ton of YouTube now. I’m always watching. Because I’m always dissecting what people are doing, like how are they editing, what kind of shots they’re using, how’s the light placement, what’s the colour grading on it. I pick everything apart when I watch. I watch a ton of different YouTube channels. He does not. Patrick does not. Patrick, what’s our editing process?\nPL: I set up my camera, and I put on my mic, and I turn on Camtasia, and I record. And then I upload it to OneDrive.\nPK: And then what happens?\nAS: And, then it magically shows up on YouTube.\nPL: I’ll give you an example. Marco was in Atlanta a couple of weeks ago. So I say: ‘Marco, let’s record a video together’ and it was something he did, and he was like ‘Yeah, let’s do it’. So I’m at the Microsoft office where I have my stuff, we set up, we recorded the video. Before I left the Microsoft office, I uploaded all of our audio files and video files, and we use a screen capture, upload it there. This was a Tuesdays night. My videos come out on Wednesday. That morning I get to\u0026hellip; So I get a text from Adam around 6.30-7am, it was early. Video’s uploaded. And I know, this is how it happens. This did not surprise me at all. I was like ‘Oh, cool’. So it’s there and I can see it, but it doesn\u0026rsquo;t release until 11.\nAS: It (videos) drops at 10:05 am Central time.\nPL: So I get to the office, and I go ‘Marco, you have to look at this’. And I show him, he goes ‘It’s done already?’ I was like ‘That’s just how it works’. I upload it, and something happens overnight and it shows up. “This is amazing, really amazing”. I do this stuff and it just shows up. So the editing process happens by me uploading, then it just magically appears on YouTube. It’s magic. I don’t know what happens between there.\nPK: So what tool do you use for editing?\nAS: I use Adobe Premiere and After Effects for the actual editing. We use Camtasia to capture the screen record, but I do no editing in Camtasia at all.\nKN: Yeah, it’s an awful application. I tried to work with that and\u0026hellip;\nAS: It’s better now than it used to be. When I started doing it, they only had a 32-bit version, they didn’t have 64-bit. And I started getting fancier with my stuff. So I end up in Camtasia with like 17 layers, and it would just crash on me.\nKN: 17?\nPL: He’s really picky. He’s really, really picky. Really picky.\nAS: There’s 6 layers by default.\nPL: Shopping for a new camera, cause my autofocus and all this\u0026hellip; He’s like ‘you gotta get a new camera’. I was like ‘I’ll get one’.\nAS: He finally bought a light. That did wonders. That did magical wonders.\nKN: Basically Adam is the technical guy.\nAS: I’m the tech guy.\nPL: Adam is the director, the producer, the editor and\u0026hellip;\nKN: You are the actor basically.\nPL: I am the actor.\n[laughter]\nAS: He’s the entertainer in chief. No, Patrick does a lot, I mean we talk our ideas, we collaborate a lot.\nPL: We talk every day.\nAS: Patrick is the demo master. Seriously, he will whip together a demo, he will spend\u0026hellip; I’m like, ‘I got to think about this demo’, he’s like ‘Let me go play with it’. Next morning he’s like ‘Man, I’ve only got like two hours of sleep, but I got this demo working’.\nPL: So the first SQLBits that I was supposed to come to, I remember this. And something happened, there was an accident in London, somebody ran into me. It was not a good thing, and my wife was like ‘You cannot go to London’. The whole family was coming, I’m like ‘But I’ve got to go, I’m doing a pre-con, I’m doing a full-day workshop’. And I had spent like months building this stuff out, and so I go ‘oof’, so I email them and it was like ‘Can you find a replacement?’, I go ‘Of course I can’. I got a guy, I know a guy, right? So I called Adam and I was like ‘Can you go?’ and he’s like ‘Of course I can go, I’m going to Bits, yes, of course I can go’. So he goes to Bits, and he comes back, he goes ‘Dude!’.\nAS: That was the most magical thing ever. I’m like ‘Dude, that stuff was just laid out and I just opened it up and I’m like ‘I can do this, this is great’. I’ve got nothing to do’.\nPL: What I did was, because there’s 8 modules or something like that. So there’s a step by step, PBIX files. If you come in in the middle of the day, you just open the .pbix, and you catch up\u0026hellip;\nKN: Even I could do that.\nAS: Cause I get worried because normally people, it’s a week before, and I’m like ‘Yeah, I didn’t actually do anything yet, can you\u0026hellip;?’ I just opened it up, oh my gosh, dude. That was amazing.\nPL: I don’t edit, though. No editing.\nAS: But he’ll do the demos.\nPK: We wanted to ask this question later, but as we’re talking about\u0026hellip; it’s going with the flow. So whenever you are in a session or in a video, there’s a lot of humor. How do you do it? Do you talk to each other before or it just happens?\nKN: How do you prepare for that?\nAS: We don’t.\nPL: You know, we’re high level.\nAS: We’ll say like bullet points, like ‘OK, we’ll touch on this, this and this, and that’s it’. We don’t talk about how we’re gonna do it. Like the actual interactions we have, that’s all on the fly.\nPL: It’s just natural.\nAS: We just gel like that. This is why I brought him on the Guy in the Cube.\nPL: I think if we practice, it messes stuff up.\nAS: So today was a good example when we were recording with Alberto, cause I do a little more, I think about how I’m gonna edit stuff, so I’m like ‘We’re gonna do a cool thing in this video’. I’m not gonna say what it is, you got to watch it. It’s gonna come out in a few weeks, I don’t know when this is dropping, but it will be out in March (icon-video-camera Using DAX to format a list of values in Power BI Desktop). So I had this thing in my head, I’m trying to describe how it is and Patrick is just sitting there like ‘I don’t do it this way’. So like ‘Wait, what do you want me to say?’ I’m like ‘Just say that!’.\nKN: I’ve heard this story yesterday, during the Power BI Manchester meeting. I basically wanted to ask this question. When did you join the team?\nPL: I don’t even remember now.\nKN: What is the story basically?\nPL: I don’t remember when I joined.\nAS: The story of how he joined. I started in 2014, after about 2.5 years is when I asked him to come onboard, but before that he just started getting into Power BI. We had known each other beforehand, but I wouldn’t say we were like good friends or anything like that. We’d gone drinking one or two times at PASS Summit or something like that. Then he just started calling me and just asking me questions about Power BI. I’m like ‘Cool, what’s up?’ He was asking good questions, and we were talking about it and then I got to the point when we were like talking every morning for like 30-45 minutes.\nPL: Every morning.\nAS: Every morning. That was like our normal routine at that point, which we still do today.\nPL: More now.\nAS: Our wives have come up to like our offices like ‘Are you on the phone? Hi Patrick, Hi Adam.’ Our families know each other. And then, after a while it was just like ‘Dude, you should do the Guy in a Cube videos with me’. He was like ‘No. No.’\nPL: I need people. I need the audience, I feed off the audience. So it’s not natural to me to sit in front of the camera and do the video, because I need you, guys. You know, the audience.\nAS: He’s much better with the audience.\nPL: I feed off of your facial expressions and your body movements, I see people get restless, it’s time for a joke, it’s time for an excursion, it’s time for a story, you know. Cause by just sitting there by myself was kinda like ‘Who am I talking to?’\nAS: So what he’s gotta do. I’ve gotten there too, like my wife watched one of the videos, and she knew what was going on that day, she saw the video, and she’s like ‘You faked that, didn’t you?’ I’m like ‘Oh yes.’ She’s like ‘Cause you sound like you were just normally like cheerful’. ‘Oh yes’. So we gotta get Patrick, cause I edit Partick’s videos and I can tell because we talk all the time, so I know what’s going on in his life, he knows what’s going on in mine and I’m just like ‘Patrick maybe shouldn’t record today’. You just wait a day. Grab a beer and then\u0026hellip;\nPK: Nice.\nAS: We refer to each as our second wife. Someone asked me ‘Which one’s the wife?’ I’m like ‘Depends on the day’.\nPL: I don’t know how he does\u0026hellip; I can’t keep the energy level. I can have a bad day and go present in front of a hundred of people, and you never know I’m having a bad day. If I have to record a video, he will know I’m not having a good day. If you watch the videos, you can tell.\nAS: Some people I don’t think they pay that close attention, they may not know. Some of them you definitely can notice. Some of them it’s not as obvious, but like I know.\nPL: And if I don’t like the topic too much too.\nAS: He never likes the video. He says this every time, he’s just ‘Oh, man, it’s just not good’. Like whatever.\nPK: But that’s how you can get better.\nAS: We don’t scrap that video either.\nPL: He won’t. And some of them are like ‘We should just not do this’.\nAS: Yes, please, don’t do this.\nPL: Yeah, but after I upload it, it’s out of my hands.\nAS: But then you call me after, it feels like ‘Man, that looked good’. We can cut it sometimes.\nKN: Tell me how you learned this ‘Yooo’. That was funny.\nPL: That’s kind of, there’s a whole story.\nAS: It’s a two-part story. So Patrick’s starts to do it and I have the same issue, and like honestly, I think it was maybe a year ago when we started getting comfortable with actually watching the videos ourselves. So we go back and watch older ones and I’m like ‘I can’t watch this, it’s awful’. We’re like Mr. robot (like a robot) and no personality, and like\u0026hellip; So Patrick’s doing this and I’m like ‘Alright, Patrick. I brought you on because you’re like\u0026hellip; your charisma, you’re just a naturally funny guy, you entertain, I wanted the energy. That’s part of the reason why I brought you on’. And I’m like ‘Patrick doesn’t show up until half way. We’ve got to fix this’. He’s like ‘Alright, what do I do?’ Alright, so here’s the trick I’ve learned on the video stuff. Get to the point where you’re really embarrassed and it’s silly and you think this is stupid, right? The minute you get to that level, you’re like doing something, ‘No, this is stupid’, you want to pull back, and I’m like ‘Don’t pull back’. That’s right where you wanna be, and it comes across on the camera great. And he’s like ‘Alright, I may just do it. OK, I’ll figure it out’. He’s like ‘Alright, I got it.’ I’m like ‘Where, what did you do?’ He’s like ‘Just watch the video’. So I watched the video and I just fell out of my\u0026hellip; I was crying. Called my wife, I’m like ‘Crystal, Come here, watch this’. She’s like ‘Oh, my gosh’.\nPL: My daughter actually came up with the whole idea. My daughter was like ‘Dad, what do you say to your friends?’ What did you guys say when you were younger? And she was like ‘Cause you are old. You’re born in the 1900s, dad. And I’m born in 2000s, right?’ And I go ‘We say ‘yo’‘. She’s like ‘Say that’. I’m like ‘say ‘yo’‘? ‘You make it fun’. ‘OK, so I’ll say ‘yo’‘.\nAS: And then it just became a thing. We start all of our presentations with the ‘yo’. We do it like this, say it like you need it, ‘dude, just do it randomly’, I’m like ‘no, I start everything with the ‘yo’, it’s a brand thing. It’s a brand.\nKN: Yes, it is, it’s a recognizable brand basically right now.\nPL: The funniest thing with the ‘yo’ for me was I was walking in Seattle, after we started doing it. I was walking on the street, I think it was at the PASS Summit. Oh no, was that the first\u0026hellip; it was called Data Insight Summit. And I was walking down the street and this guy was on the other side of the street, it was a group of them, they were like ‘Yoooo’. I was like ‘what’s this?’\nAS: So it’s funny, whenever we reach out to someone like on teams or whatever, we always start it off with ‘yo’. I was actually talking to my manager the other day, he was like ‘Hey, do you have 2 minutes’. I’m like ‘Yea, what’s up?’ And he’s like, he calls me on the phone and he’s like ‘Are you alright?’ I’m like ‘Yeah, fine, why?’ He’s like ‘You didn’t say ‘yo’?’ And I’m like ‘Oh, no, it’s alright, sorry, I didn’t even think about it’. I was like in the middle of five things and he’s just like ‘I just wanna make sure that you’re not\u0026hellip; That you’re OK, like do you need anything?’ So I was like alright, when your manager starts\u0026hellip; That’s a thing. And then the funniest thing for me was\u0026hellip; I went to a user group thing in Houston and this nice old lady, southern bell kinda thing, she’s just like (in his best southern accent) ‘Oh my Lord, that other gentleman you do the videos with. I just started watching his video, and he did the scream and just jumped out of my monitor and scared me to death. But bless his heart, you just keep doing it’.\nPK: That’s so touching. It’s so beautiful. When I heard that first ever I was like ‘wow, that’s so sweet’.\nPL: It catches you off guard, though.\nKN: I’m very happy that that was recorded.\nAS: The other reason why I loved it, and I knew this was gonna do it for Patrick and does it for both of us now. And does it for our sessions that we present live as well, with the audience. Because we do it\u0026hellip; for Patrick, what helped is he does the ‘yo’, it gets the energy level up. And now he’s carrying it through the rest of video. And then in the sessions, we start it off with the ‘yo’ and everyone’s awake and we are like ‘Alright, let’s do this’. So everyone’s mind is now in it.\nPL: These guys are bananas.\nKN: It changes the approach completely, you know what you’re doing in the next few minutes or hours.\nAS: You gotta have a thing.\nPK: It’s a great way of branding and bringing the\u0026hellip; It’s a nice icebreaker.\nPL: It is a good icebreaker.\nAS: From a branding perspective there’s a great book out there called ‘Primal Branding’. The book talks about there’s like 7-8 components of the brand, one of them is rituals. You have to have rituals of a brand. So like Nikes and Apples, they all have their thing that you can identify with. And so for Guy in a Cube the ‘yo’ is a ritual thing, the way we end our videos is a ritual thing. We’re consistent on a lot of those things.\nPL: Head over to my laptop.\nAS: The video today, we didn’t do that with Alberto, and I’m just like ‘Wow, time out’ and he was like ‘What?!’, and I’m like ‘You didn’t do your thing! That’s a ritual for you, people expect it’. He’s just like ‘Oh, you’re right’. So little things like that. We do it repetitively, cause people expect it. The minute you don’t do it, like my manager, I didn’t do the ‘yo’ and he’s like ‘What’s up, man? You’re alright?’\nPL: It was so funny. At the end Adam says ‘Or smash it if you so desire’. So I tried to do that in one of the videos and it didn’t work.\nAS: And then in the video it’s typical Patrick ‘Alright, Adam, just cut that out’.\nPL: Just cut that out.\nAS: No crap, I’m gonna cut that out.\nPK: I know the answer for this, but just for the audience. It is your thing, right? It’s not like a Microsoft thing? It’s your thing.\nAS: We obviously have support from\u0026hellip; Everyone knows we’re doing it. Even when I first started doing it, I ended up being in Redmond just for a trip to visit with a Product Team. It was right when James Philips came onboard, and I actually set up a one on one with him, I think I had a hundred subscribers at that point, so I had no audience. And I just went to, I was like “Hey, just FYI [For Your Information] I started this YouTube thing, it’s called Guy in a Cube, I’m doing Power BI videos where I’m teaching people things and help them with support stuff”. And I originally thought Marketing would shut us down because we’re not necessarily\u0026hellip; We just pick up our own topics, no-one tells us what to do. Sometimes like Marketing or Engineering team will come to us and say ‘Hey, can you do a video on this?’ And then we’ll do that, we’ll put that on the Power BI YouTube channel, not on the Guy in a Cube channel, it will brand all of that Power BI, and then\u0026hellip;\nPL: We usually don’t say the ‘yo’.\nAS: Yeah, we’re PMs on the Power BI team as opposed to just Guy in a Cube. Everyone’s cool.\nPL: It’s fun too.\nKN: Is this a secret why you added this label for some time, ‘Microsoft employee’, in your videos? Do you have to?\nAS: We have to. It’s a Microsoft policy, a social policy. You have to identify that you are a Microsoft employee. Although most people don’t notice it, it’s in every video. Cause people would be like ‘You work for Microsoft?’ I’m like ‘In the video’.\nPL: Obviously you’re not watching the videos.\nAS: And also like on the Twitter handle Guy in a Cube it says Adam and Patrick at Microsoft Power BI team. So we identify that we are Microsoft employees, we have to. It’s company policy.\nKN: OK. So while we’re talking about the videos\u0026hellip; So the next question, basically I asked this several minutes ago. How long does it take you to record 15 minutes of video?\nPL: You know, it takes me longer to put the demo together than to actually record the video. To record the video it takes me 10, sometimes, it depends on the topic, but average about 15 minutes. Then he cleans it up, I don’t know what happens after that. But no more than 20 minutes to record the video.\nKN: So the preparation for the demo takes more time?\nPL: Sometimes the demos like\u0026hellip; I just did one on Tooltips on Perf Tuning, it took a little bit. It probably took me 4-5 hours to build out the demo, because I had to create two different reports, set up DAX Studio, and I actually walk through the whole video.\nAS: We validate everything before we actually record it.\nPL: And then I’ll email someone, then wait for them to reply back or\u0026hellip;\nAS: Cause we want to make sure we know some of the questions, so we’re trying to anticipate them. This is how it works, this is\u0026hellip;\nPL: We’ll talk to each other. I’ll say ‘Oh man, do this’ and then Adam will say ‘Oh, you should do that, or don’t do that, say this’. And sometimes I listen, sometimes I don’t. And then like, we have one, like the last one I did, I wanted to have it like\u0026hellip;\nAS: That’s when he’s not being the wife.\nPK: Like a good wife. Sometimes you listen and sometimes you don’t.\nPL: Cause I wanted to have like Power BI and DAX Studio right next to each other. So when I open the report, you could actually\u0026hellip;\nAS: I told them, I was like ‘Alright, so do it this way, like you’re gonna open up DAX, you’ll do it the way you’re gonna show it first, right, and then stop, and just re-record the part, the other side of the screen that you want and then I’ll put them together, right? So just only talk over one of them, and then just redo the other part, and I’ll do the magic.\nPL: So that video took about two hours to record because it was eight takes trying to do that. And I just was like\u0026hellip;\nAS: He gave up.\nPL: Forget it. I didn’t say that, I said something else. And I just said I was just gonna record the video the way I wanted to record it.\nAS: It came out good.\nPL: 18 minutes. That video is amazing. Because the way he did it\u0026hellip; Oh, it’s amazing. You gotta watch.\nAS: Cause he had the full DAX. He had both of them. They were visible on the screen at the same time, so I just took both segments individually and blew them up side by side, and then put slower/faster underneath so you can see what the difference was.\nPL: Yeah, it’s good, I’ll show you before we pack. It’s amazing. (icon-video-camera Power BI Tooltips and report performance)\nPK: Do you have any plans of doing a video about how you edit?\nAS: I actually have a plan for a whole series. I have it mapped out, I’ve been like wanting to do it.\nPK: Like a behind the screens.\nAS: It takes so much time to do because we do four videos a week now, and you know, there’s other life stuff going on that it’s\u0026hellip;\nPL: Four videos. That wasn’t my idea. I was doing one video\u0026hellip;\nAS: So first two weeks of January, we only had our normal three videos, and I call him and I\u0026rsquo;m like ‘We’re going to four videos’. He’s like ‘What?!’\nPL: Because I’m used to doing just one video a week.\nAS: Alright, we’re doing four week, man, and he’s like ‘Alright, what’s the other one?’ And I’m like ‘We’re gonna do two-minute Tuesdays, you’ll just stay on Wednesday, and then mine will go to Thursday’. He’s like ‘Okay’ and I’m like ‘look, just either one of us, we’re just bounce for whatever we got’. And it’s been working.\nPL: They’re easy, the two minute ones are so easy.\nAS: It’s been working now. Originally I was thinking I always had this fear originally, and I think a lot of people do, whether they’re blogging or video or whatever. It’s that fear of like ‘I’m gonna start doing this, but like I’m gonna run out of ideas’. And I hit that originally when I started doing this, but I got past it. You got to push past it, and then, with the two-minute Tuesday I had that same thing creeped into me for like a split-second, I’m like ‘Oh my gosh, we’re gonna fail at this’. And he was saying things like ‘Dude, are we gonna figure this?’. And now we have like a list of like 20 of them, like a backlog, this is not a problem now.\nPK: But how do you balance it, you both have families and a day job, and all these things. There is no such thing as balance, but how do you do it?\nPL: You know, honestly, the only thing that disturbs work in Guy in a Cube is family. That’s the only thing that takes precedence, right? My little girl or my wife. ‘Hey Dad, I need’. Everything stops. Right? But if they don’t need, if nobody’s complaining, you know, that I’m down there on the Saturday recording videos, we’ll record videos, you know. But if they say ‘Hey, we’re going to go do this’. Completely stop everything and go do what they want to do. Otherwise, you know, because most of the time I’m home, you know, or my wife’s out shopping, the kids are with their friends and usually I’m by myself at home, so why not record a video instead of watching TV, when I settle down on my couch watching football, or whatever it is, I might record videos. Instead of bingeing Breaking Bad or something else.\nAS: Game of Thrones.\nPL: Let me go record a video.\nAS: You did Sopranos for a little while.\nPL: Yeah, did Sopranos, right.\nKN: Who is a fan of Game of Thrones?\nPL: Ohhhh…. Mmmm..\nAS: I can list off the dragons.\nPL: So all the stuff that\u0026hellip; Instead of just lying around, enjoying yourself, not that I don’t enjoy doing the videos, I really do. I just say ‘You know what, let me do this first and whatever time’s left I’ll go’.\nAS: Here’s the thing I found when I was in support I hit this, and then I carry it through with The Guy in a Cube stuff as well. People ask me ‘So in support you had Support Engineer, then you can get up to this title of Escalation Engineer, right?’ You’re in the Escalation team, you’re in the next level and people would ask me like ‘How do I get to Escalation Engineer?’ And I’m like ‘What are you doing besides the casework?’ ‘No, that’s what I do’. I’m like\u0026hellip; ‘You got to do a little more than that. Like how are you standing out, how are you\u0026hellip; Like what are you doing outside?’ ‘I don’t have time for that, I’m just working cases’. And I’m like ‘OK, well what do you do when you first get into the office?’ ‘Why? I work on cases’. And I’m like ‘Why don’t you carve out a half hour to work on like a blog post or something like that, right?’ ‘Well, no, I got to work on my case.’ I’m like ‘They can wait a half hour, it’s not going to be the end of the world, right? You just manage your time’. And then they’re also like ‘Why do I need to do extra things?’. I’m like ‘Okay, well what were you doing Saturday morning?’\nPL: Sleeping.\nAS: ‘Well, you know, I had to watch the game’. And I’m like ‘Okay, well, then that was the choice that you made’. Which is perfectly fine if that’s what you want to do and that makes you happy, go for it. But if you want to get ahead or you got to do these things to do that, then like\u0026hellip; So for an example. We came out here for SQLBits, my wife’s coming up on Sunday, we’re going to spend a few days in Scotland next week. We still have videos that have to go out. So what do I have to do? I told him, I’m like ‘Dude, the Friday before I’m leaving, you have to have three videos in the folder before I leave’. Saturday morning I recorded four videos and then the rest of Saturday and Sunday I had to edit eight videos.\nPL: And I’ve got an extra video.\nAS: \u0026hellip;by the end of Sunday, so that Monday morning I was flying out to England. And so now this whole week and things are dropping, people are like ‘Oh my gosh, you just\u0026hellip;’ We were walking and Patrick’s like ‘Holy crap, our video just dropped’. I’m like ‘Yeah, it’s all scheduled’. So like this weekend, next week that’s all done already, but I had to spend 16 hours on Saturday and 16 hours on Sunday, and my wife understands. Like we did, I fit in family stuff in there too, and we just make it work. I normally am up at 5 am every morning, even on the weekends.\nKN: 5 am even on the weekend? Wow.\nAS: Yeah, got to get stuff done.\nKN: I need to learn something from you. I’m trying to wake up at 5:30 or 6am.\nAS: I have an amazing body clock. I haven’t used an alarm in like 20 years. Even here, so like coming out here it’s different, you know, coming over to England from the US, I had to power through it, I sleep through it, I go to bed at like 8:30/9:00 in the morning. The next morning, yesterday morning up at 5:00 am. In the clock. Ready to go.\nKN: So we know that you are speaking a lot, yeah? Among a lot of different conferences, around the world basically.\nAS: We don’t speak at all the ones we get asked us to speak.\nPL: No way, no how.\nAS: I actually spoke a lot last year. I had a strong talk with my wife and a strong talk with my manager where we said ‘Yeah, we got to back that off a little bit’. It was too much. I agree, it was too much. So we changed that.\nKN: How much was too much?\nAS: Pretty much the whole month of September and October I was gone.\nPL: It was insane.\nAS: It was 8 week period that included three trips to Europe.\nPK: Last year?\nAS: Yeah, last September and October.\nPL: It was crazy.\nAS: It was between Denmark, Netherlands and then Spain. And those were\u0026hellip; So I came back to the US in-between those. Because I was at other conferences in the US, because it was Ignite, so it started with Denmark Power BI World Tour, then it was Ignite, then\u0026hellip;\nPL: Was there a SQL Saturday BI in there?\nAS: There was SQL Saturday Atlanta. And then Ignite, so I flew from Atlanta to Ignite in Orlando, and then from Orlando to Netherlands for Techorama, and I was back for like four days. Then I flew to Madrid for SQL Saturday Madrid. It was crazy.\nPK: I remember, this was wild. How could you do it?\nAS: And then I was back from that, and then where did I go? I went somewhere else after that.\nPK: Like Mexico, Brazil?\nPL: We went to Uruguay. It’s just too much. Honestly, I like to do more, but\u0026hellip; You don’t have time.\nAS: We get asked to do a lot of stuff, and we want to do it, we love doing it. We just don’t\u0026hellip;\nPK: It’s not easy. What was your best trip?\nAS: Uruguay was pretty fun. It was such a small group, that was their first SQL Saturday. It probably wasn’t the right thing to do from a business and impact perspective. It was an incredible trip and the guys down there that organized it, they took care of us. They were with us the whole time translating and it was great. What else? Like SQL Saturday Holland and DataMinds Connect, those guys took care of me. He does not do as much international stuff.\nPL: I’m a domestic kind of guy.\nAS: We’ve had a good time everywhere we go. Ignite was fun. Cause they buy out Universal Studios for the night, so that was fun.\nKN: Do you have any photos?\nAS: Yes, I do actually. My older daughter loves Harry Potter, and she was mad at me. She found out I was going to Universal, and she’s like ‘You can’t go’. And I’m like ‘I have to go. I’m speaking at the thing. They’re making me go. I’ve got to go’\nPL: You didn’t have to go.\nAS: And, so I’m there and of course then I’m sending her pictures ‘Hey look, here’s me with the dragon’. And she loves dragons, and she’s like\u0026hellip; I remember her face. So we’re actually\u0026hellip; because I’m going to Orlando again in a couple of weeks for the Gartner conference. And, so I said ‘I’m sorry, I’ve got to go back to Orlando again’. And they’re like ‘Nooo’. So actually I’m taking the family, we’re gonna go a few days before we go to Universal Studios. Because I was like ‘There’s gonna be a trip to some park, I know that’. And they were like\u0026hellip; My wife was not happy, and we were like ‘No, we’re going’.\nPK: Next time when you’re in the UK, I live right next to Warner Bros studios. You’re more than welcome to come and see.\nAS: Next time I come to London I will do that.\nPL: We want to go to Warner Brothers studio.\nAS: We have fun at every single one.\nPL: Everywhere we go. Bits last year was fun, the party was amazing.\nAS: (SQL)Bits party is always the best. I brought the family to (SQL)Bits, because it was in London, so they\u0026hellip; That was their first time out of the country, so we did that. And then they came to the party too, and I told them ‘Is this alright? Because I’m bringing my kids’. ‘No, no. It’s great. They’ll have a good time’. They loved it. My wife was actually mad though, because they did a tourist thing with the 9 and 3/4 thing where you got to pay like\u0026hellip; it was like 40 quid or something like that to take a picture. And then they had the same set up at the Bits party where you can\u0026hellip; She’s like, ‘son of a\u0026hellip;’ like\u0026hellip; ‘I could have just done it right here’. And you could have done it with Harry Potter.\nGuy in a Cube Unplugged. 21st February 2018 in London at CodeNode PL: And I’m not just saying that because she’s sitting here, but the user group last year\u0026hellip;\nAS: The user group last year. That was the best user group I’ve ever been to.\nPL: Next level user group. I’m just saying that cause she’s sitting over here.\nAS: Seriously, I still talk about that user group. The facility was amazing, the lighting was great, I got some epic bureau. I got a picture that I haven’t sent to Prathy about a year.\nPL: They ran out of pizza last year.\nAS: They did run out of pizza.\nPL: But somebody took us to get steak. I’m not saying that cause she’s sitting here next to me.\nAS: What I loved the best was cause we were getting ready and I was joking around, she’s like ‘Do you need anything?’ I’m like ‘Beer would be good’. ‘Boom, here you go’.\nAll Power BI guru together. 21st of February 2018 in London at CodeNode Patrick LeBlanc at London Power User Group PL: So I’m not saying anything cause she’s just sitting here but that was one of the best user groups.\nAS: And the room was packed.\nPK: Yeah, it was like 300-ish people. It was silly. I mean I enjoyed it. The way you guys started and the way you’re talking was just like\u0026hellip; I don’t know how many people like you\u0026hellip;\nAS: So that’s an example of like\u0026hellip; We don’t really talk to each other. I didn’t necessarily know what he was gonna do, he didn’t know what I was gonna do.\nPL: We just knew we were gonna do some demos. It’s like ‘wait, who’s gonna go first?’\nPK: I think it’s obviously whatever we see on the screen, it’s like\u0026hellip; We can see there’s a lot of natural talent. It’s great. We enjoy it.\nSteak dinner, London. KN: I still remember our dinner after the event.\nPL: It was great, the steak was amazing.\nAS: I didn’t know where we were going. We went downstairs somewhere, underneath the road\u0026hellip;\nPL: Somebody took my coat and then we sat down. Oh, so good\u0026hellip;\nKN: We know that you have this amazing flow during a session. Do you prepare yourself for a speech?\nPL: For the sessions that we do?\nAS: It depends. Some of it depends on where like some of the stuff we’ve done, we’ve done before. For that, we don’t really prepare much. We know what we’re gonna do. Some of it we’ll maybe do a little update and touch-ups because of new features and things of that nature. If it’s not new, we’ll spend more time on it. But not to the level that I know some people do where they have to rehearse it. We don’t do it.\nPL: So slides. I hate slides.\nAS: I’ll throw some slides on that.\nPL: I’ll kind of put some stuff together.\nAS: He’s the demo guy.\nPL: I hate slides. We have a session tomorrow.\nAS: He’s great. I told you, he’s the demo master.\nPL: We have an intimate, a new one.\nAS: We have a whole lot of stuff.\nPL: We have this new one called Intimate with Desktop.\nAS: We need to sit down and talk about that. Because I’m not so sure what we’re doing there.\nPL: All the demos, it’s great, it’s phenomenal, and it’s not your typical\u0026hellip; You know, it’s just different things.\nAS: We’re showing things.\nPL: The people may not have used.\nAS: And maybe not even know about.\nPL: You may not even know about it. You may not even know that it’s there. There’s not one slide, there’s a ‘Hey, this is us’ and ‘Ask questions’. And then all the .pbix files are on OneDrive somewhere, we’ll give you a link. Check them out. Now there’s a new one that we do on paginated reports. I did spend some time, putting some slides together, haven’t given it to him yet. They’re awful.\nAS: We’re doing a keynote tomorrow. We actually need to spend some time a little bit on that. We wanna make sure that’s a little more\u0026hellip; We’ll get it sorted.\nPL: So not a lot of preparation unless it’s something we’re doing new. And then most of the time\u0026hellip;\nAS: But even then, it’s not like\u0026hellip; We don’t do like rehearsing timings. We don’t do any of that.\nPL: Somebody will put it together and go ‘Okay, this is where you’re gonna talk, this is where you’re gonna talk’.\nAS: Cause the other thing like we know, like timings, like we know our timings. And we know that we can stretch it or pull it back. We know how to do that. So we don’t worry about that. We never worry about that.\nPK: I think when you know the content it’s kind of\u0026hellip; It makes it easy.\nPL: But then it’s like I know when to interject almost intuitively. When to say\u0026hellip; Or I almost can feel when Adam wants to say something, cause I’ll slow down and I’ll go ‘You’ve got something to say? You want to add something?’\nAS: We just know each other. It’s a natural thing.\nPK: I did presentations with other people, it’s not easy.\nPL: It’s not easy, it is not easy. I refused to present with other people. I refuse. I’ve been asked a couple of times. I think the last time was a couple of PASS Summits ago.\nAS: I’ve had to it before and then also they start doing what we do, and it’s like ‘No’.\nPL: I’m just not doing it. I’ll just present by myself. No, I’m sorry. I’ll say ‘No, no, thank you’. ‘Well, you can’t come to the conference’. ‘That’s fine. I go to enough.’ I’ll record some videos at home.\nAS: I think that’s part of why The Guy in a Cube works. Because of the natural flow.\nPK: So Patrick, it’s a question to you. You are a great presenter, I’m sure you know that.\nAS: Don’t tell him that! Cause there was time where his head was so big he had to walk sideways into the room.\nPL: I gotta get serious.\nPK: Who do you look up to?\nPL: To be honest. This is true and I’m not gonna say that, because he’s sitting in the room. Adam. I really, really do look up to Adam. Adam’s been\u0026hellip; So I was in Sales at Microsoft and it was\u0026hellip; I’m not gonna say this. It wasn’t that great, right? The role, because I’m like this. If they invited me in to talk to your company, and I’m with this guy. This is not knocking Microsoft, right? And the guys selling them Data Lake, and I go ‘oh, they don\u0026rsquo;t need Data Lake\u0026hellip; you just need a SQL Server’. ‘You need Azure SQL Database, the smallest one’. And I’m like ‘Oh my God, I gotta\u0026hellip;’ And then they’ll come back ‘Patrick, you just cost me, you know, a million dollars’ I’m like ‘Yeah, but it’s the right thing to do for the customer’. Every morning I was like ‘Oh God, I gotta go and talk to these people and I gotta listen to this’. ‘It’s OK, dude. We’re gonna get you out of that job’. I say ‘Okay’. Then I started doing Guy in a Cube. People say a lot of stuff about Patrick, he’s not technical enough, he’s not\u0026hellip; ‘Don’t worry, dude, don’t worry about it’. Right? In the end. Made it happen.\nAS: It wasn’t me. I mean, he was doing the work.\nPK: We all can see it in the videos and everything.\nPL: Even on the team I don’t feel like I belong, but he goes ‘Don’t worry about it’. Just keep doing what you’re doing. So there’s other people, right? So honestly one of my biggest idols is President Obama, right? I’m a black man, and he was the first brown president, right? And for other little brown boys, who are facing a lot of challenges\u0026hellip; In America, he’s a role model because it’s not who he was as a president or what he did as a president. He became the president, right? I don’t care what he did or how bad he did or how good he did. He became, so now I can say ‘Wow, it’s possible’ for young brown men, too. And Will Smith. I’m just kidding. Adam had a story about it.\nAS: It has been multiple occasions where he\u0026hellip; Last night it happened, where it was just like: ‘Has anyone told you, you look like Will Smith?’ I’m like ‘Oh, here we go’.\nPL: No, I was just kidding. Because you gotta think about, when we work together, Adam or I go ‘Man, how to do this’. And he goes ‘Hang on a second’. And he will send me a link. So he’s not like ‘I’m not gonna tell you how to do this, go figure this shit out yourself’. So sorry, beep that out. Go figure it out. Cause he knows I’m gonna figure it out. I may have some questions, you know, but\u0026hellip;\nUseful links Guy in a Cube: WebSite | YouTube | Twitter | Facebook | Instagram | Snapchat\nAdam Saxton - Twitter\nPatrick LeBlanc - Twitter\nProducts: Microsoft Power BI | Ideas\nEvents: SQLBits Website \u0026amp; Twitter | Techorama | Power BI World Tour\nCommunity: London Power BI User Group | Meetup\nUpdate:\nRecommended book: Patrick Hanlon - Primalbranding: Create Zealots for Your Brand, Your Company, and Your Future\n","date":"2019-04-26T06:30:15Z","image":"/uploads/2019/04/ASF021-SQLBits-with-GuyInACube_1200x775.jpg","permalink":"/2019/04/asf-021-guyinacube-interview-part-1/","title":"ASF 021: Guy in a Cube interview (part 1)"},{"content":"Hello Data Folk! There is plenty of readings for this week, so grab a coffee and let\u0026rsquo;s start digging in!\nPress Smarter, faster, safer: Azure SQL Data Warehouse is simply unmatched\nMeet three additional enhancements that continue to make SQL Data Warehouse even better today.\nUse ADF Mapping Data Flows for Fuzzy Matching and Dedupe\nFuzzy Matching with Spark Soundex function in ADF sounds interesting!\nDevOps In Azure With Databricks And Data Factory\nSample project with whole CI/CD example code in Azure DevOps.\nSQL For Cosmos DB – Tips and Tricks\nCollected by Adam Aspin.\nAzure Data Explorer\nWatch what\u0026rsquo;s that data analytics service and how it can help in your scenarios.\nCreating a custom kubectl plugin to connect to SQL Server in Kubernetes\nAndrew (T) showed how to write kubectl plugin from scratch.\nExecuting Packages in an SSIS Catalog in a Container\nAndy Leonard (T) briefly presents how to run SSIS packages using PowerShell.\nBI Survey 19\nChris (T) reminds us that the largest annual survey of BI users in the world happens again. Take part in the survey.\nDownload and install sqlpackage\nA new release of sqlpackage is now available on all platforms, fixes a number of reported issues. Version 18.2 (Build 15.0.4384.2)\nASF 021 video invitation (Star Wars Theme)\nCheck this trailer of the next episode of #AskSqlFamily podcast.\nA word from Microsoft’s CEO Climate change is one of our world’s most pressing challenges, and today we’re announcing new steps to reduce our environmental impact and help our customers build their own solutions to create a more sustainable planet. https://t.co/IFHEFRhToX\n— Satya Nadella (@satyanadella) 16 April 2019 Video: Power BI Get Data: Import vs. DirectQuery vs. Live https://www.youtube.com/watch?v=y4S-k34qmnw\nSmile Corner ","date":"2019-04-21T22:52:19Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/04/last-week-reading-2019-04-21/","title":"Last Week Reading (2019-04-21)"},{"content":"Since now, we will not keep our guests in secret. We want you to be prepared and wait if you\u0026rsquo;re interested in.\n2 vs 2 interview: Adam, Prathy, Kamil, Patrick at SQLBits. First time 2 vs 2 in #AskSQLFamily podcast. I had a big pleasure to led that chat with Prathy. Another huge pleasure was having those Guy In a Cube\u0026hellip; I meant, in the podcast.\nDo you want to know why actually YouTube? What tool Adam does use for post-production? Why the very first movies were like \u0026ldquo;awful\u0026rdquo;? Who has been scared to death and why? What were the circumstances of joining Patrick to the team and what\u0026rsquo;s fun fact related to his name? And finally: how did the YOOO become a hallmark - get know the entire story about it.\nThat gonna be MEGA! Spread the words about the upcoming episode which gonna be published next Friday (26/04).\nThanks!\n","date":"2019-04-19T00:48:37Z","image":"/uploads/2019/04/ASF021-video-invitation.jpg","permalink":"/2019/04/asf-021-video-invitation-star-wars-theme/","title":"ASF 021 video invitation (Star Wars Theme)"},{"content":"Good morning guys and gals! This time many updates and articles from ADF field, so keep reading once you sit comfortably with coffee in a hand.\nPress Handling Schema Drift in Azure Data Factory\nIt\u0026rsquo;s actually a video where Cathrine (T) presents Schema Drift feature.\nDevOps In Azure With Databricks And Data Factory\n(Almost) everything you need to know if you are interested in all these technologies (like me).\nDynamic File Names in ADF with Mapping Data Flows\nSee how to spread out the stream into many files. Explained by Mark Kromer (T).\nSend Notifications from an Azure Data Factory Pipeline – Part 2\nHow to work around the lack of sending email activity?\nWhat if the Actual Execution Plan was always available for any query?\nDon\u0026rsquo;t you believe in that? Awesome news, isn\u0026rsquo;t it? Pedro Lopes (T) explains more.\nHow to develop and submit Spark jobs to SQL Server Big Data Clusters in IntelliJ\nAzure Toolkit for IntelliJ supports SQL Server Big Data Cluster Spark job development and submission.\nA word from Microsoft’s CEO It's here. Whether you're developing, testing, debugging, or deploying—make every keystroke count with #VS2019. Read the blog: https://t.co/lD9z4Kgx5F pic.twitter.com/TTSeDXGm3V\n— Visual Studio (@VisualStudio) 2 April 2019\rVideo: Azure Data Factory SCD Type 1 with Mapping Data Flows Smile Corner Nothing funny I saw last week. Did you? Wanna help? Please do! Send a message on the Twitter, add #smilecorner tag and @SQLPlayer. Let\u0026rsquo;s have fun together!\n","date":"2019-04-08T00:05:04Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/04/last-week-reading-2019-04-07/","title":"Last Week Reading (2019-04-07)"},{"content":"Hello SQL/Data folks! The first week after returning from MVP Summit (Redmond, USA) was murderous for me. What about your past week? In this post, as every week, I review the news about Microsoft Data Platform. Join me.\nPress Three Ways to Use Power BI Dataflows\nMelissa (T) shows up 3 options for Dataflows and explains when they might be considered.\n(video) Deploy to Azure using GitHub Actions\nGopi joins Donavan to discuss how to deploy to Azure using GitHub Actions, which helps you to configure CI/CD from the GitHub UI.\nAzure.Source - Volume 72\nLast week\u0026rsquo;s updates from Azure (services)\u0026hellip;\nMicrosoft Azure portal March 2019 update\n\u0026hellip; and from portal itself.\nAnnouncing the public preview of Data Discovery \u0026amp; Classification for Azure SQL Data Warehouse\n\u0026ldquo;Your code is your documentation\u0026rdquo;, they said. It\u0026rsquo;s important to have the data classified.\nWe have moved!\nSQL Server Tiger Team has moved its blog to the new place HERE.\nShould we use stored procedures or queries built in the app?\nBrent (T) discusses the scenarios when queries from an app might be reasonable.\nData in Devon – Saturday 27th April session schedule\nLess than a month left to Data in Devon (UK) conference - check training deep-dive sessions.\nThe first time when the podcast has been video recorded and I was hosting that talk along with Marek.\nA word from Microsoft’s CEO Intelligent manufacturing is now a reality, and the advances we announced today spanning end-to-end IoT security, digital twins and more will help our customers capture the new opportunities this era will create. https://t.co/AWE4YmSU1y\n— Satya Nadella (@satyanadella) 28 March 2019\rVideo: 5 ways to get Power BI Support #pbifamily Smile Corner ","date":"2019-03-31T20:49:11Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/03/last-week-reading-2019-03-31/","title":"Last Week Reading (2019-03-31)"},{"content":" Introduction CTO at Clouds On Mars, former Microsoft employee as Data Insights Product Manager for Poland. In 2007 Pawel started Polish SQL Server User Group (PLSSUG), currently known as Data Community Poland, an official PASS Chapter in Poland. Pawel has been a speaker at many conferences in Poland and worldwide (e.g. SQLDay, SQLSaturday, European PASS Conference). Six times Microsoft Most Valuable Professional (MVP). Father of three. Avid runner.\nThis talk occurred during SQL Saturday #824 in Krakow (Poland) on 9th February 2019 (Saturday).\nInterviewers: Kamil Nowinski, Michal Sadowski.\nDo you know which conference was inspired by the SQLBits conference?\nHow to make a success as a team at work?\nWhat’s important in your public speaking?\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript KN: Hi, Paweł.\nPP: Hello, guys.\nKN: Thank you for accepting the invitation to this podcast.\nPP: The pleasure is mine.\nKN: Cool. At the beginning could you tell us what your name is and where you live?\nPP: OK, my name is Paweł Potasiński, I live in Warsaw, Poland for my whole life.\nKN: Cool. What are you doing for a living?\nPP: I work as a CTO [Chief Technology Officer] at Clouds on Mars Company, it’s small\u0026hellip; You can call it a start-up. We are the partner of Microsoft, focused mainly on polish market, but I’d say spreading more over Europe and also having some of the businesses in US.\nKN: Which part of Europe you’re focusing on as well?\nPP: Usually in those well-positioned countries like Switzerland, the Netherlands.\nKN: What kind of position do you have there?\nPP: My job is to keep an eye on technologies. To let’s say, drive directions for the whole company, in which technologies we should create our solutions and our products. We’re not focused on solutions, we try to figure out some products. Mainly related to analytics, BI [Business Intelligence], AI [Artificial Inteligence] stuff. I have a 5-people team, they’re called ‘SPEC Team’. You can call it something like special forces in the company. All those guys are architects with different skills, so I have a big data guy, I have an analytics guy, the guys who are skilled with SQL Data Warehouse and stuff like this.\nKN: Do you focus only on Microsoft solutions?\nPP: Yes, pure, 100% Microsoft partner. Currently, we are not heading towards any different vendor.\nKN: You have been working for Microsoft for several years. Tell me more about your experiences from that period of time.\nPP: It was a good time actually. I started in 2012 I guess and worked there for 4.5 years. So my first role was something called Partner Technology Advisor. This role doesn’t exist anymore, it’s now called Partner Technology Strategist, I guess. By that time the job responsibilities were to take care of the partners, recruit partners, to work on the specific Microsoft technologies. My job, as you may imagine, was to capture different partners to work with SQL, to believe that SQL Server is the best database platform in the world. After 3 years I moved to the marketing role. It was Product Manager on SQL Server as well. And then it was called Data Platform Product Manager. So then my responsibilities were to drive strategy on the Polish market, how to position Azure and SQL Server, Power BI among different competing technologies.\nI’m not saying it was a bad moment in my life, it was definitely inspiring, I’ve learned a lot, but I just decided that it’s time to move forward, because I really enjoyed being close to technology. Believe me, in Microsoft, if you’re not a consultant in terms of being PFE [Premier Field Engineer] or engineer or you don’t work in Redmond, then there is a good chance you’re just beginning to become more and more like a sales guy. So I just didn’t want to\u0026hellip;\nKN: Losing touch with technology, yeah?\nPP: Yeah, just getting rusty.\nMS: Let’s move a few years back. How did you start working with SQL Server?\nPP: I was doing my studies. There is a guy, his name is Włodzimierz Dąbrowski, you may remember this name. He’s a PhD at Warsaw University of Technology. He started my journey with SQL Server, it was with SQL Server 7.0. I did my master thesis, it was somehow related to SQL Server, I analysed several web-based technologies, how they worked and cooperated with SQL Server. That was my beginning. And then I had a job which partially was related to database administration. It was not just SQL Server, but also some PostgreSQL databases, that’s how it started. At first, I was a DBA you can say, then I moved to T-SQL developer and trainer. And finally, I landed in the BI land, so I started to deliver Data Warehousing and BI. Somehow people associate me with this, but I really have more to do about T-SQL than MDX and DAX.\nMS: Looks like a very long journey, but also a very interesting one. You are the founder of the Polish SQL Server User Group. And you’ve been president of that organization.\nKN: Named Data Community Poland.\nMS: Exactly. Can you tell us a few words about that organization, what was the driving force behind setting it up?\nPP: I think that the roots of this organization, this organization right now, formal association\u0026hellip; Originally it was not like that, it was a group of people just meeting and talking about SQL Server, purely colleagues, friends. We didn’t want to make it formal actually, but we observed that this time when we meet each other and talk to each other about SQL Server, share our observations, practices, maybe some codings, is a well spent time. It was a good investment. So there were a couple of people that over that time became familiar like Maciej Pilecki. He was already a well-known and famous speaker. Marek Adamczuk, Radek Kępa, you may know those guys because they somehow became well-recognized experts in Poland and sometimes over the world like Marcin Szeliga for example. So we just figured out that it could be a good idea to make those meeting more official and maybe bring some more people onboard, because, as we spread among different companies, we observed that there are lots of, I would say, smart people working hard with T-SQL and SQL Server, knowing some details that may be just forgotten and even not observed by other guys. So, the purpose of Polish SQL Server User Group was to bring those guys and share their knowledge.\nMS: You’re presenting around the world and I would like to ask about the child of the Polish SQL Server User Group, so SQLDay Conference. How does it compare to other international events?\nPP: I’d say it looks very good. Of course, at the beginning, we had to be inspired by bigger conferences like SQLBits in the UK, definitely one of our inspirations. But over time I think SQLDay became something that is a very, very famous brand across Europe and probably over the world. By inviting guys from Redmond, from the US, I think, you guys are doing a good job in spreading the word. I think we, as an association, are recognised also anywhere you go and ask about SQL Server and Microsoft Data Platform in general. So yeah, good job.\nKN: True, the SQLDay event is very recognizable not only in Poland. Last year the tickets were sold out 2 months before the conference started.\nPP: Definitely what people like about SQLDay is the community that somehow happened here, it was created for 3 years. Probably our Polish hospitality is also what is recognised, because we are really nice people for the speakers, for the attendees from different countries, so yes.\nKN: You’re also speaking a lot, not only in Poland, but also at different, other different conferences.\nPP: Now it’s not like that. You have some old data. I’m not travelling around the world anymore, I just focus rather on Polish conferences locally, but it’s just because I’m a family man, so it just takes time to travel. Right now, after having the third baby, I’m just a bit more of a domestic guy. So you may meet me here, at SQL Saturday, at SQLDay definitely in Poland. I consider going to some neighbourhood countries, maybe Czech Republic or Slovakia, maybe Germany, but it’s not the time. I have to wait a while for my youngest kid to rise.\nKN: Yeah, because we didn’t mention at the beginning that we’re sitting right in the SQL Saturday in Krakow. After this interview, you are going to present a session. OK, so the next question is how do you prepare for a speech?\nPP: Well, over time it became a bit easier. 3 years I learned that. The most important thing about your session is how clear you are in your speech. It’s about setting some goals just at the very beginning, so start with ‘Why are you presenting this session?’. Then bring some facts to the table, never rely on documentation or things that were created by somebody else, which you probably haven’t tested in any serious project. If you have real-world stuff to show, it’s definitely easier to bring something to the attendees. And also right now I became a bit lazy, because usually, I have a co-speaker, which makes things double easier. So each time we just spread the work between two people. So at this moment, I have a session with my friend from Clouds on Mars, Paweł Kruczkowski, big data architect in my team. He’s responsible for some of the demos in our today’s presentation.\nMS: You have mentioned that you are a father of three kids. And also I know that you are an avid runner. How do you prioritize those things together?\nPP: That’s a very difficult question. Definitely the very first number is family. That’s why I rarely have time to be a part of any races or attend some foreign conferences. That’s changed with the third kid. The family is first, running is definitely the last one. So, I think, work and community is still number two and running is something that is just present in my life, because I just found out years ago that it may save my life actually. If you’re getting heavier and heavier there’s nothing good about your health, so I really encourage people to start doing something to become healthy. It can change your life and make you feel better.\nKN: So how this work-life balance works for you?\nPP: It doesn’t work. There is always some area that suffers from others. So now, when I have an intense time at home and at work, running suffers a lot. Let me give you an example by numbers, SQL guys, we love numbers. In 2015 I ran 3500 km per year, which makes it like 10 km per day. Right now I run maybe 1500 a year, which is over twice less than 4 years ago.\nKN: Still very good, I think.\nPP: I know, for people who don’t run it seems a lot, but it means that you run three times a week, 10 km each run. So it’s not that bad, but definitely less than in my history.\nKN: I was running, maybe not as much as you, but still, it’s for me\u0026hellip;\nPP: You look like you’re running.\nKN: Some of your activities are on your technical blog. How do you find a time, inspiration for the posts?\nPP: You asked me the killing question, because my blog is dead for some time. I promised myself to start over and keep on blogging, but it’s so difficult. Especially that some of the activities planned by myself always are around my work. So, I think I may be visible rather on some blog or video-blog of my company in the next couple of months than on my own blog. Sorry, but this blog is more a memory than something that is actual. You can find some historical entries like things around historical versions of SQL Server.\nKN: Yeah, but finger crossed for your New Year resolution.\nPP: Yeah, that’s a resolution, definitely, yeah.\nMS: You’ve been working for Microsoft and you’ve been awarded an MVP award. What do you think about the MVP program those days?\nPP: I think it’s unique and actually I think we are fortunate to be MVP on Data Platform or AI because some of the guys here are also AI MVPs. The reason why we are fortunate is that because the products groups that are built around Data Platform of Microsoft seems to be unique. The contact, the intensity of the contact with those product groups while I’m an MVP is very unique. I mean, I can ask practically every single question and I’m answered, let’s say, in a few hours usually. So the contact with other MVPs, the contact with product groups is something that brings the most value to this program. The network built around the world, definitely.\nKN: We’ll be out of networking in March, MVP Summit.\nPP: Definitely I’m going there.\nKN: Me too.\nMS: What is the personal achievement that you are most proud of?\nPP: Losing 30 kilos. And also convincing my wife to have the next child after, over 10 years of break.\nKN: You’re talking about the third one?\nPP: Yeah, the third one is now\u0026hellip; It was actually not over 10 years, the third is 2,5 while my sons are 14 and 11, so that makes a huge difference. It’s tough to be a newly-born parent again, because it seems like it’s the first one, it’s the first girl we have. That’s something different.\nKN: Yeah, that was the challenge, I can imagine.\nPP: You have to remember everything that you’ve done in the past, so yeah.\nKN: Yeah, I already forgot all those things from the beginning.\nPP: But it’s definitely worth that.\nKN: And about the kilos. You lost those kilograms many years ago I think, yes?\nPP: Yes, I started running in 2012, I guess. I started my job at Microsoft, so I decided to at least lose some kilos, because definitely that was a stressful moment for me. I just stepped into an international company with a different culture, different, I’d say, speed of work. That was something that came to my mind, so I decided to do something for myself. And I was running and I don’t regret that. It was a good time. And I continue that with maybe not with as many successes in terms of beating my personal records, but still running.\nMS: Which part of your work is the hardest one in terms of being a CTO?\nPP: Definitely it’s people management. You may imagine how hard it is if you have 5, maybe more employees to manage. Each one is a senior guy, each one with different skills. Probably the biggest challenge is to switch from the “answerer”, as I call myself, and I usually was the guy who answered all the questions. Now I have to bring the answers out from all those guys to all the people who need answers to our technical questions. When I talk to my team, I usually say ‘Guys, my success is to make you smarter than me. If we prove to our colleagues at the company and our clients that you are smarter than me, then we’re done. Then the job is done’.\nKN: We will win in this situation\u0026hellip;\nPP: I observe how those people work every day. I think they get more and more convinced that there are some areas that I just didn’t cover. Because there’s no time to follow all the technologies. Especially now, when you have not just SQL Server, the single version, but you have Azure with hundreds of services and open source world coming in. So we just divide all those technologies among many people, that makes things easier, definitely.\nKN: What hints would you give to young people who want to start working in the IT market?\nPP: Build your network, definitely, so SQL Saturday, SQLDay, the conferences that fit your interests are definitely something you should attend, no matter what. Meet people, talk to people. There are lots of open people who will advise you, who will talk to you about technology. You don’t have to be an expert. You should pick 1 or 2 technologies which you should stick to and be an expert. Try to make this decision wisely, because you may see that some of the technologies devaluate quickly, so that’s a crucial decision for you. But also do not close yourself for different technologies. I’m not just talking about Microsoft. If you get familiar with different technologies from different vendors, you’re one step ahead, because you know that your clients may be not that fascinated about the Microsoft world, because they’ve seen something different.\nKN: And then we are going smoothly to the next question. Microsoft did very well in Azure SQL Data Warehouse compared to its competitors in this area. So either from a performance perspective, as well as from a financial perspective. Have you had a chance to work with other products in this area, comparing them?\nPP: Actually as I moved towards analytics and data warehousing, I rather played with other Big Data engines, PaaS [Platform as a Service] Big Data engines, so I had some benchmarks against RedShift and Google BigQuery. So that was not actually pure relational engines. About what you’re saying on SQL Database. Actually, if you look at SQL Database from a SQL Server admin perspective, a developer perspective, there is something that is really meaningful, I mean, those databases are used around the world, but still I think there are some little and bigger things that can change in Microsoft’s offering. What I mean by that is, for example, we’ve seen moving from DTUs [Database Transaction Units] to vCore last time. Most of the newest versions of SQL Database, I mean Managed Instances, Hyperscale and stuff like this, they rely on vCore. However, if you compare, how vCore relates to Memory, then you’re not so happy. I don’t find it a very, I’d say, attractive offer to have one vCore corresponding to 5.5 gigs of RAM. There’s still lots to do, but yes, it’s definitely a good direction that the well-known and a really great SQL Server engine is moved to the PaaS services. And if I compare what I’ve seen about the Big Data and Warehousing tools, Microsoft definitely did a good job also by providing well-known interfaces like T-SQL language everywhere they can. If you observe the newest announcements, yesterday they announced the general availability of Data Explorer in Azure. It’s custom. It’s also based on the assumption that you might bring some T-SQL skills, some T-SQL queries and then operate on really Big Data stuff, doing thing in a matter of sub-seconds. So very cool stuff.\nKN: And for all the orchestration processes. What do you use in the company?\nPP: We actually stick to SQL Server Integration Services. Eventually moving towards Azure Data Factory with some of the use cases addressed by Databricks. We look forward to Data Factory, Data Flows and Mapping Data Flows, but for now yes, SQL Server Integration. Mostly because we got used to all the goodies like BIML stuff. However, what we see nowadays is that because of the slow start, sorry to say that, but slow start of Data Factory, some of our clients are using, for example, Air Flow to operate ETL processes and data orchestration.\nMS: You’re working a lot of cloud products as you have mentioned, like Azure SQL Data Warehouse. From your experience, how many customers are ready to migrate to Azure from on-premises to cloud?\nPP: Actually I’d say it’s easier to start new businesses and new projects than migrate. Migration is always a pain because there are lots of legacies. In Polish companies, you may find definitely lots of legacy systems, which are hard to migrate because of many things. Logic spread among databases, different old technologies. Lots of factors actually. It’s usually a big project to migrate something and you may not be satisfied by the argument of just having pure cloud. But if you’re going to start a new project, the pace, the fast start of the cloud and the fact that you may operate with the cost, which is just another part of your optimization, is really something that people are just deciding more and more to get into the cloud.\nKN: Do you observe that more companies are going to migrate?\nPP: Yes.\nKN: Yeah, it’s absolutely easier to start doing new product, a new solution in the cloud.\nPP: Actually Clouds on Mars is the first company that working for this company I have an impression that I have more cloud clients than on-prem clients. So definitely we have like 70% of cloud deployments over on-prem. But even if our client deploys on-prem, it’s always a matter of discussion and maybe providing some additional arguments to make them consider cloud as the next step. It’s changing. I remember SQLDay three or four years ago when we were sitting there at the discussion panel and Adam Machanic was mentioning that he has some Fortune 100 clients and none of them is moving to the cloud. I think Adam should visit us now and let’s revise this question, because it changed over time. I had the same impression over that time. It’s just a couple of last years that the cloud exploded. It happened.\nMS: As you have a lot of experience with the cloud, what do you think about the “cloud first, cloud only” that starts to become a buzzword?\nPP: It is a buzzword, because more and more clients are asking about how you address vendor lock-in problem. So I think in the nearest future we may observe a trend like companies focused on Data Platforms will have to bring on board people who are specialized in administration of infrastructure. By infrastructure, I mean infrastructure as a code, so pure templates, some containers stuff, things like that. Because not everybody will rely on PaaS Services. People are thinking about PaaS Services as ‘OK, so this project, if I step into this service, there will be a problem if I decide to get back, let’s say, migrate to a different cloud vendor for example’. So that’s always a question.\nKN: What do you think about the current trend of migration to the cloud? I mean, is it going to replace all those on-premise servers in the future?\nPP: No, I think SQL Server still has many, many different, unique offerings like, for example, Master Data Services or the newly created with [SQL Server] 2019 Big Data Clusters. It’s not like it’s going to finish, but it’s going to be more hybrid. I think we observe, let’s say, a gain of stronger connection between the on-prem world and the cloud just to make the migration easier of course.\nKN: How do you adopt DevOps approach in your company?\nPP: It’s hard. I could point you to many, many feedback items at Microsoft Azure forums. We try to do that actually, we see that there is a good direction, because practically every single team at Microsoft Product Groups sees that there is a problem if we’re going to perform DevOps and CI/CD [Continuous Integration/Continuous Delivery]. Then without the appropriate tools, we’re just lost. I’m referring here to, for example, project templates for (Azure SQL) Data Warehouse, which are still missing. They are in preview, we are testing it, we’re providing feedback, but it’s not a mature product. We need that, we need also maybe some additional tools for lightweight clients like Azure Data Studio to make things more efficient to work as developers. I really invite people working with pure .NET application, because they have everything in place. But with databases, if you develop a SQL database, you’re covered, because you have a very, very good product in database project template. SSDT [SQL Server Data Tools] is not perfect, but we try to cover every single missing feature with third parties but it’s definitely there. I think the biggest problem now is to make CI/CD and DevOps with Power BI. It’s a different topic. I think they’re heading in the right direction, but still, the major things are missing there.\nKN: Yeah, I think this is a low priority for Microsoft right now.\nPP: I think it’s on high priority, but it’s difficult because of how Power BI was implemented at first. I think it’s a kind of technological dept to be paid right now.\nMS: Where we can find you? You have mentioned about the blog.\nPP: Actually I’m present usually on the Facebook discussion group of Data Community Poland, so you may find me there. Sometimes I tweet, if I remember well it’s @PawelPotasinski. From time to time I’m present at Azure communities, not just Data Community meetings, so you may also find me there. But as I mentioned, I’m a rather busy domestic family guy at the moment. Now also with my new role at the company, there’s not much time for me to perform some activities. I’m just trying to focus on the most important stuff like SQL Saturday, SQLDay. That’s why I’m here.\nKN: Perfect, thanks.\nMS: Thank you.\nPP: Thanks a lot.\nUseful links Paweł Potasiński - Twitter Account | LinkedIn Profile | Clouds on Mars\nSQLDay Conference - website | Twitter\nData Community Poland - Facebook | Website | Twitter\n","date":"2019-03-29T07:05:48Z","image":"/uploads/2019/03/ASF020-SQL-Saturday-Krakow-with-Pawel-Potasinski-1200x630.jpg","permalink":"/2019/03/asf-020-pawel-potasinski-interview/","title":"ASF 020: Paweł Potasiński interview"},{"content":"OMG! My brain almost blew up. All because of MVP Summit I\u0026rsquo;ve just returned from.\nObviously (due to NDA), I can\u0026rsquo;t tell you what I\u0026rsquo;ve seen and found out, tons of very interesting sessions, not only including those about Big Data Cluster. Believe me or not - the universe is expanding. This year is becoming even more interesting for me now.\nIn the meantime, let\u0026rsquo;s take a look at what\u0026rsquo;s new and publicly available.\nPress Using docker named volumes to persist databases in SQL Server\nQuick Andrew\u0026rsquo;s (T) post explains how to keep the databases automatically attached.\nVisual Studio 2019 Launch Event\nLearn more about that coming product.\nGenerating a Jupyter Notebook for Glenn Berry’s Diagnostic Queries with PowerShell\nDiscover the new possibilities of Azure Data Studio along with well-known scripts.\nT-SQL Tuesday #112 – Roundup\nShane O\u0026rsquo;Neill (T) summarizes the topic about a cookie jar.\nSQLBits – A World of Free SQL Server and Data Information Online\nDr Greg Low (T) opinions about that event (with a link to whole content).\nNow available for preview: Workload importance for Azure SQL Data Warehouse\nA CEO can have a higher priority with classifying requests.\nA word from Microsoft’s CEO Today we introduced new capabilities to make collaboration more inclusive and intelligent, while delivering new levels of security and compliance, for the more than 500,000 organizations using Microsoft Teams. https://t.co/084Bnj9KM8\n— Satya Nadella (@satyanadella) 19 March 2019\rVideo: Global Azure Bootcamp 2019 Smile Corner ","date":"2019-03-25T23:12:46Z","image":"/uploads/2019/03/IMG_6054_MVP_coffee.jpg","permalink":"/2019/03/last-week-reading-2019-03-24/","title":"Last Week Reading (2019-03-24)"},{"content":"New podcast\u0026rsquo;s episode is coming (next Friday) and in the meantime, take a look you can watch that episode\u0026rsquo;s trailer.\nDo you know which conference was inspired by the SQLBits conference? How to make a success as a team at work? What\u0026rsquo;s important in your public speaking?\nSimply click the PLAY button above or move to the YouTube: https://youtu.be/zdxp4dByH9c\nCould you guess who is the next guest?\nEnjoy!\n","date":"2019-03-23T15:57:28Z","image":"/uploads/2019/03/ASF020-video-invitation.jpg","permalink":"/2019/03/asf-020-video-invitation/","title":"ASF 020 video invitation"},{"content":"It is the fourth post on series related to installation and configuration of SQL Server client tools required.\nThis series is for professionals who start their journey with SQL Server administration and also for those who want to extend and structure their knowledge on SQL Server administration.\nPreparations We are starting with the following assumptions:\nAll steps described in this first blog post has been configured ISO file with SQL Server binaries and Cumulative Update files are available - steps how to download it are described in the second blog post There is a connection between Windows server and domain controller No issues with connectivity with Active Directory domain (no AD-related errors in System Event Log) SQL Server instance is installed as described in the second blog post or third blog post As a first step, we are going to download SQL Server Management Studio.\nHow to download SQL Server Management Studio? Open your favorite web browser and go to Download SQL Server Management Studio (SSMS) website\nScroll down to latest version of SSMS section and click on the Download SQL Server Management Studio link:\nOnce the file is downloaded run the installer:\nOn Welcome screen review License Terms and Privacy Statements. Once it is done you can start the installation by clicking on Install button:\nWait a few minutes till Management Studio is successfully installed:\nOnce SSMS is installed, click Close:\nStart SQL Server Management Studio from Start Menu and connect to SQL Server - I am using a local installation of SQL Server.\nOnce the connection is successful you should be able to see the following connection\nYou can also use Azure Data Studio that is compatible with other Operating Systems like macOS or Linux.\nHow to download Azure Data Studio? Open your favorite web browser and go to Download and install Azure Data Studio website:\nScroll down to User Installer and click to download an installer\nWait for download till complete and run installer\nThe installer should start - click Next\nRead and accept the license agreement\nSelect destination location for binaries\nSelect Start Menu Folder\nChoose additional tasks\nOnce everything is ready, click Install to proceed with the installation\nWait till installation completes\nOn the last screen leave Launch Azure Data Studio option enabled\nAzure Data Studio should start. To have access to all new features that are in a preview state, click Yes on Would you like to enable preview feature\nProvide Server name and click Connect - I am using a local installation of SQL Server\nOnce the connection is successful you should be able to see the following connection\nSummary Management tools are key for efficient and fast administration of the database systems. This blog post extensively describes how to install both tools provided by Microsoft.\nLinks More information about steps described above can be found on the following sites:\nDownload SQL Server Management Studio (SSMS)\nDownload and install Azure Data Studio\n","date":"2019-03-21T08:01:53Z","image":"/uploads/2019/03/office-594132_1280.jpg","permalink":"/2019/03/installation-sql-server-client-tools/","title":"Installation SQL Server client tools"},{"content":"Hi there! I\u0026rsquo;m in Bellevue, WA today when finishing this post. My first MVP Summit and the second trip to the US ever. A very exciting week is coming for me and I wish the same for you.\nPress SSIS Catalog Dashboard\nTim Mitchell (T) has shared the code on GitHub.\nUsing Kubernetes Deployments for Updating SQL Server\nLearn how to switch between various version of SQL Server in seconds.\nPower BI Desktop March 2019 Feature Summary\nAmanda (T) is presenting the next set of features in Power BI.\nThe Absolute Guide To Selecting Between Azure Analysis Services And Power BI Premium\nCheck out Dave\u0026rsquo;s (T) report which helps you decide which service suits you better.\nAzure Data Factory: From Databricks Notebook to Data Flow\nLet\u0026rsquo;s migrate some transformations to Mapping Data Flow in ADF.\nWhat sessions do you want to see at GroupBy next month?\nIt\u0026rsquo;s too late for voting for a session but find out what kind of sessions happens on the 11th of April.\nSSIS interview questions\nHey recruiters, here you have some example interview questions for SSIS developers.\nA word from (former) Microsoft’s CEO I was lucky to be a young person when the digital revolution was just getting under way, and my Microsoft colleagues and I had the chance to help shape it. Celebrating #Web30 leaves me feeling nostalgic about 1995. https://t.co/krIFxTGd1q\n— Bill Gates (@BillGates) 12 March 2019 Video Microsoft Azure South Africa datacenter regions are now generally available\nhttps://www.youtube.com/watch?v=DGbS4qIyh7E\nSmile Corner ","date":"2019-03-17T14:44:57Z","image":"/uploads/2019/03/IMG_5994_Bellevue.jpg","permalink":"/2019/03/last-week-reading-2019-03-17/","title":"Last Week Reading (2019-03-17)"},{"content":"Press Databricks-Connect - FINALLY!\nDo write your code in VS Code and run it remotely on Azure Databricks cluster.\nAutomatically validate E-mail attachments with Flow and Power BI\nSend an email if upcoming attachments don\u0026rsquo;t meet criteria. Automatically altogether.\nUse XEvent Profiler to capture queries in SQL Server\nIf you still use SQL Server Profiler - check this Erin Stellato (T) post.\nSearching Complex JSON Data\nPlaying with JSON format does not have to be tough.\nAzure SQL Data Warehouse Query Store\nForce plan is not yet available but generally looks promising.\nA word from Microsoft’s CEO This #InternationalWomensDay, we're celebrating the incredible contributions of women to our industry and also working to remove barriers to ensure that the next generation of female innovators joins their ranks. #MakeWhatsNext https://t.co/sGeQ1LhfCV\n— Satya Nadella (@satyanadella) 8 March 2019 Video: Rerun activities inside your Azure Data Factory pipelines https://www.youtube.com/watch?v=raql6x5AAlc\nSmile Corner ","date":"2019-03-11T00:16:59Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/03/last-week-reading-2019-03-10/","title":"Last Week Reading (2019-03-10)"},{"content":"SSDT project has something we call Variables. On the Internet (posts, forums) they also are known as placeholders or tokens, especially when talking in a deployment/CI/CD/DevOps context. Variables can be found for each SSDT database project: Right-click on the database project and choose \u0026ldquo;Properties\u0026rdquo;, afterward go to \u0026ldquo;SQLCMD Variables\u0026rdquo; section.\nThey are particularly useful when having more than one database project in a solution, but not only because the variables might be used in many variations and scenarios.\nPredefined SQLCMD variables For each project, we can use some variables already exists and hardcoded the project, which are:\nDefaultDataPath DefaultFilePrefix DatabaseName DefaultLogPath Obviously, they can not be (re-)defined again on your list.\nWhen might use variables? There are a couple of specific cases when variables are particularly helpful, although, the list is not limited to these:\n(Other) Database References Linked Servers Login/users Variables for T-SQL code Define environment name/code Database References First of all, import all the databases you need to have into the solution. It\u0026rsquo;s obligatory to create a database reference when you have 3-part object references in T-SQL code. An imported code might look like this:\nSELECT * FROM SecondDatabase.dbo.Customer This is fine for SQL Server Engine, but it is not valid in SSDT project and as a result, you\u0026rsquo;ll get error or warning. What needs to be done then? You must add a database reference to [SecondDatabase] and replace ALL its occurrences in SSDT code using a newly created variable:\nSELECT * FROM [$(SecondDatabase)].dbo.Customer Once you create a new database reference - related variable will be created automatically for you.\nCreate new database reference (menu) \u0026ldquo;Add Database Reference\u0026rdquo; (window) In \u0026ldquo;Database Reference\u0026rdquo; you have 3 options to be chosen:\nDatabase projects in the current solution - this is the option we want to use referencing the SecondDatabase. In the list, you see all database project present in current solution, except the current database. System database - use this option when you must add a reference to [master] or [msdb] Data-tier Application (.dacpac) - this allows you to make a reference to a compiled version of a database project. Generally, I do not recommend using DACPAC files inside the solution, but sometimes it might be helpful in specific scenarios. This is another story for a separate blog post. \u0026ldquo;Database location\u0026rdquo; determine where the referenced database is located in. We have 3 options over here:\nSame database - useful for specific cases, which we\u0026rsquo;ll be discussing in a future post. Different database, same server - the most-used option. Select it when a referenced database is located on the same server. That means we use 3-part object notation. Different database, different server - Choose that one when a referenced database is located on another server. In this case, you will use 4-part object notation, having 2 variables (server \u0026amp; database). Choosing one of the above option - one or more fields below becomes available. For variable fields (Database variable, Server variable), by default, values are like $(database). I\u0026rsquo;d suggest to leave it like this unless you have a significant reason to change it. Notice, that for system databases - only \u0026ldquo;Database name\u0026rdquo; field is available. We do not use variables for them.\nAt the bottom of the window, we can see a usage example applying new variable(s).\nWhen everything set up - click OK and new database reference will be created. You\u0026rsquo;ll see it on the list:\nList of references databases As I mentioned earlier here - the next step is to replace all occurrences with variables. This is the toughest one.\nAlso, be aware that although all variables would be replaced by their values which are exactly the same as they were - your code (Stored Procedures, Views, etc) change. It happens only for those references not being used brackets for server/database name.\n-- Original code SELECT * FROM SecondDatabase.dbo.Customer; -- SSDT code with applied variable SELECT * FROM [$(SecondDatabase)].dbo.Customer; -- Code deployed from SSDT to physical database SELECT * FROM [SecondDatabase].dbo.Customer; Don\u0026rsquo;t fool yourself - the whole process is not easy for bigger projects which contain many databases. The more databases and references you have - the more time you spend preparing all references to make the solution fully valid, i.e. without any errors and warnings.\nCreate DB reference Every time when creating database reference - Visual Studio create one or two variables if it doesn\u0026rsquo;t exist in the project yet.\nCreate variable manually The variables might be used in the various scenario, like:\nDefine Environment Code (or name) Login and/or user Variables for T-SQL Variables (hard-coded values) Extra variables in order to control behavior in Pre/Post-deployment scripts Others\u0026hellip; From my experience and from your votes (I have asked for what reasons do you use SQLCMD Variables on Slack - #SSDT channel) results that the second most-used reason is holding an environment code in order to control the behavior of deployment in Pre/Post-deployment files.\nPool\u0026rsquo;s results: Which cases do you use SQLCMD Variables in your SSDT projects for? This post is not aiming to describe in details all these scenarios, but let me show you one example. A good practice is to create user(s)/group in the database as they are on this level. Let\u0026rsquo;s do that using a variable.\nFirstly, add the variable in the database project with correspondent value:\n$SvcWebSite = svc_company_website\nThen, add the following rerunnable code to create user safety and grant appropriate permissions:\nIF NOT EXISTS (SELECT 1 FROM sys.database_principals dp WHERE dp.name = \u0026#39;$(SvcWebSite)\u0026#39;) BEGIN PRINT \u0026#39;User [$(SvcWebSite)] created in [$(DatabaseName)] database.\u0026#39;; CREATE USER [$(SvcWebSite)] FROM LOGIN [$(SvcWebSite)]; END EXEC sys.sp_addrolemember @rolename = \u0026#39;db_datareader\u0026#39;, @membername = [$(SvcWebSite)]; PRINT \u0026#39;User [$(SvcWebSite)] was granted db_datareader role permissions in [$(DatabaseName)] database.\u0026#39;; GO What is the advantage of that approach? Notice, that you can very easily change the value when deploying the database - having different users between environment (e.g. DEV, TEST, PreProd, PROD), for instance:\nDEV, TEST - svc_company_website_dev PrePROD, PROD - svc_company_website_prod In a further post, I describe how to deploy the database directly (sic!) from Visual Studio and how to support it with publishing profiles.\nProject file So, adding references and variable we have made some changes to the project. As we all work with source control (I believe so!) it\u0026rsquo;s worth to know WHAT kind of changes should we expect.\nThe changes due to new database reference:\nThe changes due to new SQLCMD variable(s):\nOk, so we have got these variables/tokens in the code. How the hell will they be replaced in the physical database in a SQL server? That magic action happens during a deployment, hence keep reading.\nThe entire code of this post can be found in my Git repo: icon-github nowinskik/ssdt-demo\nThat\u0026rsquo;s it in this post. In the next post, I describe briefly how to deploy such a prepared database project to a physical target server.\nThanks for reading.\nUpdated @ 08/11/2019\nThis post is already part of the whole series about SSDT.\n","date":"2019-03-06T08:41:24Z","image":"/uploads/2019/03/poker-2378914_1280.jpg","permalink":"/2019/03/tokenization-of-database-project-in-ssdt/","title":"Tokenization of database project in SSDT"},{"content":"Hey everyone! I was very busy at the SQLBits conference last week, but it did not stop me to prepare some materials for you.\nPress Data Never Rests\nVideo from SQLBits\u0026rsquo; Keynote is available now as well as the majority of other sessions!\nSQL Server 2019 community technology preview 2.3 is now available\nSome exciting features have been announced at #SQLBits: Big data clusters, Database engine, SQL Server Analysis Services (SSAS).\nNew Objects, Columns, and Messages in SQL Server 2019 CTP 2.3\nBrent almost immediately scanned new version.\nPython visualizations in Power BI Service\nBuild your own Python visualizations in Power BI which are being updated with cross-filtering in the report.\n620 million accounts stolen from 16 hacked websites now for sale on dark web, seller boasts\nI didn\u0026rsquo;t realize that the prices for stolen data are pretty low.\nSQLBits on Twitter\nCheck out all popular tweets with #sqlbits hashtag for last week.\nASF 019: Simon Whiteley interview\nDo listen or read our latest conversation.\nA word from Microsoft’s CEO Our collective opportunity in this era of the intelligent cloud and intelligent edge has never been greater, and with our announcements today we’re democratizing technology so everyone can benefit. https://t.co/NvIQM0uWfr\n— Satya Nadella (@satyanadella) 24 February 2019 Video: Microsoft shows off HoloLens 2 mixed reality headset at MWC https://www.youtube.com/watch?v=e-n90xrVXh8\nSmile Corner ","date":"2019-03-03T22:45:36Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/03/last-week-reading-2019-03-03/","title":"Last Week Reading (2019-03-03)"},{"content":"Hello folks! This time quick review about Kubernetes, containers, automatic deployment of Data Factory and few things about Power BI. It\u0026rsquo;s gonna be a tough week: SQLBits starts on Wednesday! See you there?\nPress Building A Kubernetes Cluster For SQL Server 2019 Big Data Clusters, Part 3: Big Data Cluster Creation\nI have just found this post on Chris Adkin\u0026rsquo;s (T) blog and feel that must read the whole series.\nActive Directory, Windows Containers and SQL Server\nMichal (T) presents pretty handy and not so easy solution as he could have seen at first glance.\nContinuous integration and delivery (CI/CD) in Azure Data Factory using DevOps and GitHub\nRayis Imayev (T) explains the deployment of ADF step by step.\nPower BI architecture Diagram v4\nDusty (T) has updated his diagram adding Power BI Dataflows and more feature that are or will be available.\nIncremental refresh \u0026amp; query folding\nDo verify when a folding might not happen with an incremental refresh.\nASF 019: Simon Whiteley interview\nThe latest podcast is still warm and fresh. Check out my conversation with Simon about ADF, SSIS, Azure SQL DW and other things.\nSQL Saturday #829 in Pordenone on Twitter\nI have been there last weekend. Big thanks to my Italian friends for having me!\nMicrosoft a Leader in Gartner’s Magic Quadrant for Analytics and BI Platforms for 12 consecutive years\nMicrosoft is really far away from its competitors now.\nA word from Microsoft’s CEO During #BlackHistoryMonth, colleagues across Microsoft are sharing how \"one small act\" helped define who they are today. I was moved by their stories, a powerful reminder of the enormous difference that a mentor, teacher or friend can have in our lives. https://t.co/oVvZKjGEv4\n— Satya Nadella (@satyanadella) 22 February 2019 Video: Dragonflies Turned Into Spy Drones https://www.youtube.com/watch?v=awrw3mBgSac\nSmile Corner source: DBI ","date":"2019-02-25T15:12:30Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/02/last-week-reading-2019-02-24/","title":"Last Week Reading (2019-02-24)"},{"content":" Introduction As Chief Cloud Architect with Adatis Consulting, Simon Whiteley has been working with the Microsoft BI Stack for a decade, starting off building traditional warehouses with SSIS and SSAS but bringing in alternative technologies where needed. Most recently, he has turned his head to the clouds, advocating new methods of getting insight to the right people. Whether it\u0026rsquo;s automating file ingestion into Azure Data Lakes, applying massive compute via SQLDW \u0026amp; Data Lake Analytics, or simply shifting your current SSIS packages into the cloud, he can help.\nBig believer in the wider data community - Regularly present on cloud analytics architectures and runs the SQL Surrey PASS chapter \u0026amp; co-organises the London PASS Chapter. Simon is a strong advocate of the use of modern development practices in the SQL World.\nData Geek, Cloud Herder, Speaker, Cyclist, Microsoft Data Platform MVP and general Tech-Nerd.\nThis talk has taken place after London SQL Server User Group meeting in London (UK), CodeNode on 29th January 2019 (Tuesday).\nInterviewer: Kamil Nowinski.\nHow many bicycles he has got at home and where was the destination of last year of a 400 km trip?\nWhich tool among Azure is the closest to his soul?\nWhat exactly is hiding behind the term \u0026ldquo;Modern Data Warehouse\u0026rdquo;?\nWhen Azure Data Factory (Mapping) Data Flow will be publicly available?\nAlso, we will try to answer the fundamental question: is the SSIS dead or alive?\nFind out the answers on these questions and much, much more.\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen to this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript Kamil Nowinski: Hi, Simon.\nSimon Whiteley: Hello.\nKN: Thank you for accepting my invitation to this podcast, SQL Player.\nSW: More than welcome, thank you for doing my user group.\nKN: I was happy to help as always. Could you, at the beginning, tell me what your name is and where do you live?\nSW: Okay, so I\u0026rsquo;m Simon Whiteley. I live in a place called Gravesend recently, in Kent, previously London. So I\u0026rsquo;m kind of London, but not quite London anymore. London in my heart.\nKN: It depends who you ask, they tell you that it\u0026rsquo;s not London or it\u0026rsquo;s a part of London?\nSW: No, I live really far away from London, I live in Kent.\nKN: What are you doing for a living?\nSW: So I am the Chief Cloud Architect for a company called Adatis, so a consultancy. We do advanced analytics consulting. Used to be BI developer, used to write lots of SSIS, lots of SSAS, SSRS. These days it\u0026rsquo;s all entirely Data Lakes, Databricks, SQL Data Warehouse is using that modern Azure stack, so everything I do is making sure we\u0026rsquo;re doing things in the most modern way, we\u0026rsquo;re learning and using the most recent approaches. I personally tend to move between a lot of different clients, making sure we\u0026rsquo;re sharing learnings. Making sure if we build something really cool in one place, we\u0026rsquo;re actually sharing that with the other people working with, and we don\u0026rsquo;t have two entirely different ways of doing the same thing. So I spend a lot of time across a lot of different clients just talking tech.\nKN: You\u0026rsquo;re synchronizing the works of other people.\nSW: Yeah, dipping my own in, going \u0026lsquo;Oh, no, there\u0026rsquo;s a better way\u0026rsquo;, generally annoying our consultants, that\u0026rsquo;s kind of my role.\nKN: When you said \u0026rsquo;the modern architecture\u0026rsquo;, there\u0026rsquo;s a modern of everything basically those days. What do you mean by that?\nSW: So that is the big shift of taking all the learnings from Big Data. It\u0026rsquo;s very much the idea that you\u0026rsquo;re either a Big Data developer or you\u0026rsquo;re a SQL developer and there\u0026rsquo;s a lot of different tools and techniques used between the two. So Modern Data Warehouse is all about taking those approaches, taking the Big Data ways of doing things and just making it normal, making it part of how you do BI. So every BI solution we do these days has a Data Lake, it has Elastic compute. It uses things like Parquet and avro. And it\u0026rsquo;s kind of all these things that used to be just Big Data and just making it path of just how we do things.\nKN: So there\u0026rsquo;s basically no more solution without Azure? Without a cloud solution?\nSW: That\u0026rsquo;s not true anymore. It was that way. So very much it was the\u0026hellip; If you\u0026rsquo;re in Azure we go great. You\u0026rsquo;re going \u0026ldquo;Data Lake\u0026rdquo;, you\u0026rsquo;re going Databricks, you\u0026rsquo;re going all this modern new tech. And if he came to me and said \u0026lsquo;I\u0026rsquo;m going on-prem\u0026rsquo; I would wince and go, \u0026lsquo;Okay, you\u0026rsquo;re getting SSIS, you\u0026rsquo;re doing things traditional, we\u0026rsquo;re not elastic scaling, it\u0026rsquo;s a very different way of working. 2019 has Big Data clusters which gives an in-house spark engine based on docker, which means you have a lake. You have inbuilt spark, you have all of the things we\u0026rsquo;ve now been doing and calling the Modern Data Warehouse in Azure. You can now say “I\u0026rsquo;m gonna do this on-prem”. I mean, you still have to have a bit of tin underneath, you still have to have big enough VM host to support the maximum capacity of the scaling and stuff. But all the tools and techniques, the landing things as flat file, so you can acquire data really quickly. All of that stuff you can now do is on-prem learnings when 2019 comes out, cause it\u0026rsquo;s not out yet. But when it is, there\u0026rsquo;s a big change. The way we talk about on-prem is going to change a lot this year. I sound like a Microsoft sales person. I\u0026rsquo;m just scared because there\u0026rsquo;s a lot of stuff I\u0026rsquo;m gonna have to reapply, lots of ways of working that we had kind of written off never having to think about that on-prem, that suddenly now is going to come into every single on-prem conversation. So I need to get a lot of solutions, a lot of patterns, and a lot of it is going to be taking what we do already and re-applying it. But there\u0026rsquo;s a lot of rework and lot of rethinking about how we do stuff.\nKN: That\u0026rsquo;s good, we\u0026rsquo;re still advocates of Microsoft stuff.\nSW: Absolutely. And it means I still have a job this year because everything has changed again.\nKN: Everything\u0026rsquo;s changing. So when we\u0026rsquo;re talking about technology. I know that you are doing a lot of stuff with Microsoft tools. Which one of the solutions, one of the services is the closest for you and which of these two? So the question is what is the closer to your soul: Azure SQL Data Warehouse or Azure Data Factory?\nSW: Either way, one of them is gonna get really angry with me. So it\u0026rsquo;s very weird I\u0026rsquo;m going to say this, but it\u0026rsquo;s definitely Data Factory. Because Data Warehouse is awesome, it does a good job and you can do a very, very heavy crunching. It\u0026rsquo;s kind of cool. But it\u0026rsquo;s not solving a new problem. You could always just have a really, really big SQL Server and put stuff on it. You can have Cassandra, you can have Snowflake, there are different kinds of MPP engine you can have. Whereas Data Factory now, definitely V2, definitely the current iteration solves so many problems for me. Cause all things like having to use BIML to make SSIS packages and having to write PowerShell to crank out so many iterations of ADF V1. All of the pain of creating like any kind of orchestration and scheduling in Azure it\u0026rsquo;s gone with V2.\nKN: Even in generic pipelines or so?\nSW: Yeah, you can just build a generic pipeline and then say \u0026lsquo;Now run this same pipeline with a few variables for each of my thousand tables\u0026rsquo;. Because it used to be, you could get around the problems by code automation. But at the end of day to support it you still then have a thousand separate pipelines to support and now I can just say \u0026lsquo;Well there\u0026rsquo;s a generic thing and that will do whatever I want, no matter what the data source is\u0026rsquo; and that\u0026rsquo;s key to the whole ethos of Data Lakes. I don\u0026rsquo;t want to build a separate thing for every single file, I want to make it as\u0026hellip; you want to remove the barrier to entry. I want to build frameworks and I want to say \u0026lsquo;So now if I\u0026rsquo;ve got a client who\u0026rsquo;s using ten different tables in their SQL database and they go \u0026lsquo;Actually I need another one\u0026rsquo; then all they need to do is just add a line to a database and then magically that table comes in next time. So the whole reason you use Data Lakes is to remove the barrier to entry of getting data into a system. I want to make it as easy as possible to start collecting that data. I don\u0026rsquo;t necessarily want to process it. I don\u0026rsquo;t know what logic I\u0026rsquo;m gonna apply, I don\u0026rsquo;t know how I\u0026rsquo;m going to use that data, but I want to start keeping it. And if I can just write and either add JSON file or change a database record and suddenly that works, and that means my business users can then have a power app over the top of it and go \u0026lsquo;Here are the tables I would like to keep\u0026rsquo;. And I can just push that, that\u0026rsquo;s no longer my problem. My business users define what data gets pulled into a lake, that makes my life so easy and it really gets behind the idea of data lakes. So it\u0026rsquo;s kind of like I think for the business Data Factory\u0026rsquo;s had more of a change. So Data Warehouse is neat and does a job, but the business users don\u0026rsquo;t really see that much of it. It tends to be abstracted away for business users, whereas Data Factory actually has some use for them, it has some purpose, they get a tangible return based on Data Factory. As for me, it makes a difference.\nKN: Yeah, true.\nSW: That was a long answer.\nKN: Another one is extension of Azure Data Factory, which is the private preview feature, Data Flow. What do you think about that one?\nSW: I mean, you know it\u0026rsquo;s close to my heart. Again, that\u0026rsquo;s a business user thing, so we\u0026rsquo;ve gone the whole big data processing element. We\u0026rsquo;re using a lake, we\u0026rsquo;ve got raw data landing in a lake. It\u0026rsquo;s maybe too big or it\u0026rsquo;s the wrong shape. It\u0026rsquo;s coming in a bad format or it\u0026rsquo;s coming in a streaming data and that means I can\u0026rsquo;t put it into SQL Data Warehouse. So I\u0026rsquo;m using the lake to kind of fix those problems, the four V\u0026rsquo;s of Big Data. So I\u0026rsquo;m trying to get that data and making it accessible. And Databricks is now the de facto answer. So it was Data Lake Analytics for a while and again someone who would have to go away and learn U-SQL, they\u0026rsquo;d have to learn this new language which is just used in one single tool to do it. And now Data Lake Analytics has kind of fallen out of favor, everyone\u0026rsquo;s pushing Databricks and again, you need to go into Databricks and you can write SQL in there. But you need to use either Scala or Python at some point. Maybe a bit of R, maybe a bit of SQL but\u0026hellip; Now that\u0026rsquo;s kind of been great and we started using it and, we\u0026rsquo;re using it lots of places, we\u0026rsquo;ve built lots of wrapper, kind of framework patterns. And that\u0026rsquo;s great. So I can just pass some parameters in and have it automatically process some data. But again, someone has to understand that to support it. Someone needs to understand Python or Scala and go and manage that. That\u0026rsquo;s another language I need. Whereas Data Flow means I can pass it to something like SSIS developer, who can pick that up and go \u0026lsquo;Okay, I know what this is about. This is just a Data Flow\u0026rsquo;.\nKN: I will put this component here.\nSW: Okay, it\u0026rsquo;s going horizontally, it\u0026rsquo;s not going vertically, it goes horizontally, so it\u0026rsquo;s a bit different. But otherwise it\u0026rsquo;s designed to be really familiar and similar to that kind of traditional BI user who\u0026rsquo;s finding the whole thing intimidating, going: \u0026lsquo;Do I need to rescale, do I need to retrain, do I throw away all my knowledge of the past 10-15 years of development?\u0026rsquo; And he\u0026rsquo;s going \u0026lsquo;Okay, no, this is something I get, but it\u0026rsquo;s gonna write it in Scala behind the scenes, so it\u0026rsquo;s actually going to run of natively efficiently on Databricks\u0026rsquo;. And it gives you a lot of control. I mean, there\u0026rsquo;s critics of it, there\u0026rsquo;s people who don\u0026rsquo;t like it, because it is seen as an abstraction layer, it\u0026rsquo;s convincing people: \u0026lsquo;Don\u0026rsquo;t write code, code\u0026rsquo;s too hard. Use this graphical user interface instead\u0026rsquo;. An there\u0026rsquo;s valid reasons for that. But you can drop down. For me for the simple stuff, for the business user stuff, for the data analyst level, where they don\u0026rsquo;t need to retrain, they don\u0026rsquo;t want to rescale, but they need to be able to, you know, you can have the people who understand the transformation rules designing your transformation pipelines, because it\u0026rsquo;s nice and easy. And then if there\u0026rsquo;s something really hard that is complex, or is going to need complex performance optimization, you can drop down to the Scala, you can have someone writing the hard specialist bits and not doing the \u0026lsquo;I need to knock out a thousand of these kinds of jobs\u0026rsquo;. So it\u0026rsquo;s separating the problem. You\u0026rsquo;ve got straightforward business transformation, anyone can do that, it\u0026rsquo;s really straightforward, and then your specialist and it\u0026rsquo;s kind of meaning you can balance that team out. It\u0026rsquo;s cool, it works for me.\nKN: It\u0026rsquo;s cool because you can create the business logic with Data Flow without code. And it\u0026rsquo;s very efficient process.\nSW: The thing I\u0026rsquo;ve not tried to do yet is to make the data flow fully parameterized, so I can apply the same logic to different data sets.\nKN: I tired. You can\u0026rsquo;t.\nSW: That for me is the problem, because I\u0026rsquo;ve gone so far with normal Data Factory. In terms of getting it so it\u0026rsquo;s co-driven, sort of parameter-driven. Getting it so I don\u0026rsquo;t need a thousand different pipelines to process my files. And then if I drop down to Data Flow, I\u0026rsquo;m suddenly \u0026lsquo;Oh no, I now need a thousand different pipelines to manage my files. So it does have some limitations still.\nKN: But I think this is a little difference between Azure Data Factory pipeline and Data Flows. Data Flows, I guess, it\u0026rsquo;s basically designed for specific reasons, for specific scenarios. Framework probably will not help here.\nSW: But then it\u0026rsquo;s the same in code. We can have things automatically checking data types, checking files are in the right shape, we can do all of that automatically.\nKN: Yeah, from the source perspective for example you should be able to apply the parameter, but currently you can\u0026rsquo;t.\nSW: But to do business transformation, to apply logic, nothing\u0026rsquo;s gonna write that automatically for you, no matter what the vendor promises you. It\u0026rsquo;s not gonna do it for you. So you either have a separate SQL script for each of your data sources or a separate Data Flow for each of your sources or a separate Python script. You need to have your transformations documented somewhere. So Data Flow is somewhere you can use to keep the transformations. Some people won\u0026rsquo;t want to and some people \u0026lsquo;what you want to have?\u0026rsquo;. Maybe you know kind of SPARQL, they want to have the SQL written inside the Spark engine because that\u0026rsquo;s more natural to mop out of them, they can. So it\u0026rsquo;s options, it\u0026rsquo;s all about having different tools available to you to do different things. There\u0026rsquo;s certainly not a \u0026rsquo;everything is going to happen in Data Flows from now on\u0026rsquo;. I\u0026rsquo;d say much more likely that there\u0026rsquo;ll be generic things that can be one generic Python script that runs for every single thing, just saying \u0026lsquo;what structure should it be, is it in the right shape, all of the data type\u0026rsquo;s right\u0026rsquo;. You don\u0026rsquo;t need the Data Flow to do that, cause you can\u0026rsquo;t parameterize it. So pass it to a single script, tell it what shape it should be in and that\u0026rsquo;ll run. And you can use that same notebook for every single file type. So that kind of stuff I wouldn\u0026rsquo;t use Data Flow for. And only when it is file specific, drop-down and use it there.\nKN: That might be a very strange question. What\u0026rsquo;s changed in ADF between version 1 and version 2?\nSW: All of it! The variable-driven stuff, so suddenly you don\u0026rsquo;t need to generate a ton of script, you can just have reusable components. But also the control flow stuff, so whenever one picked up ADF, and they wanted it to be SSIS, they wanted to work the same way and so many people were frustrated with ADF v1, because it wasn\u0026rsquo;t SSIS, and now the fact that you can have failure paths, you can have Foreach loops, you can have weights and web lookups and you can get reference data from a database and use its parameter. All these things, it\u0026rsquo;s so sound that we\u0026rsquo;re excited about it, because it\u0026rsquo;s just the most obvious thing you should be able to do. And that\u0026rsquo;s like the big shift between ADF v1 and v2. It became a product that actually you can use properly seriously in an enterprise situation. V1 you could barely, you have a load of workarounds, a load of code and it kind of worked. Honestly for the entire lifecycle of v1 I didn\u0026rsquo;t use it. I used it once or twice on projects where we thought we\u0026rsquo;d try and force it. And it was OK. Most the time we used SSIS, we had SSIS sitting on a VM and we were sad about it. So v2 it\u0026rsquo;s just so much more acceptable. It is a good product. I was such a hater of Data Factory for so long. It was one of those things, you mentioned Data Factory and I\u0026rsquo;d go \u0026lsquo;Ha! No-one\u0026rsquo;s going to use that\u0026rsquo;. Now I love it! As you said, I love it more than I love SQL Data Warehouse and you know that\u0026rsquo;s dear to my heart.\nKN: One of the questions during my session was when will Data Flow private preview become public, but didn\u0026rsquo;t hear your answer.\nSW: Soon. All I can say is soon. Yes, 2019.\nKN: In Microsoft nomenclature?\nSW: I can neither confirm nor deny any of these comments. But it is coming soon. It\u0026rsquo;s coming soon to the point of people who were kicking off projects now, I probably recommend they look at it, and they consider using it, because it is getting near to that point in its life cycle. That\u0026rsquo;s about as detailed as I can be in that answer.\nKN: Last question about technology. Is the SSIS dead or alive?\nSW: That was another question from tonight that I was very good and I didn\u0026rsquo;t say anything about. Do I think SSIS is dead? So it\u0026rsquo;s becoming a niche use case, so there\u0026rsquo;s this very few scenarios when I would recommend SSIS now. And it tends to be, if you have a massive team with a ton of SSIS existing, you\u0026rsquo;ve got a lot of in-house SSIS skills and all of your application exists as SSIS package, and you wanna lift and shift, absolutely use SSIS, there\u0026rsquo;s nothing wrong with it. So you\u0026rsquo;ve got Data Factory has the SSIS integration run time behind it, it\u0026rsquo;s all SSIS-packaged on it and it\u0026rsquo;ll work the same as an on-prem SSIS box. What it won\u0026rsquo;t do is take advantage of most of the whole Azure to the point of having Azure. It won\u0026rsquo;t elastically scale. So you can have multiple nodes, but all that will do, it will help with concurrency of package execution. So if you try to do one massive package, then you need to pay for a really big VM to actually get that to work properly. And it takes 20 minutes to turn on, so absolutely everything that I do these days, we\u0026rsquo;re trying to tie the amount of compute you\u0026rsquo;re using as closely to your workload as possible. So things that have a twenty-minute workloads are not tied at all to your thing. It just means that SSIS for me doesn\u0026rsquo;t scale properly and is too slow in terms of startup to actually fit in a modern data warehouse. But if you want to get that quickly and you\u0026rsquo;ve already got a lot of SIS stuff, by all means use as a lift and shift. Get your whole platform onto Azure, get it working using the resisting stuff and then it\u0026rsquo;s basically technical dare to go and strip that out and solely replace it with things that naturally scale more. But that\u0026rsquo;s all I see it for these days, it\u0026rsquo;s a stepping stone to moving to the right wing\u0026rsquo;s in things that are more cloud native. That\u0026rsquo;s all I use it for these days.\nKN: The new solution will be adopted year by year by all existing companies consequently. And if the project or your, let\u0026rsquo;s say, performance point of view needs a more robust and faster process, probably you\u0026rsquo;ll adopt the idea of SQL. If you have budget, time for that.\nSW: It\u0026rsquo;s not so much ADF, it\u0026rsquo;s Databricks. So it\u0026rsquo;s using an in-memory Spark engine. And whether you do that by accessing Databricks directly, or whether you do it using ADF as an abstraction layer, either way what you want is a Spark layer, because that\u0026rsquo;s the thing that can naturally scale out over multiple nodes. No matter how you get to that Spark layer, that\u0026rsquo;s the way you should be going.\nKN: Okay, let\u0026rsquo;s talk a little bit about private things. Not really deep private, but you know. I know that you have, let\u0026rsquo;s say, a lot of bicycles at home. Seven?\nSW: 7, technically 6, one\u0026rsquo;s my wife\u0026rsquo;s. 6 for me, 1 for the wife.\nKN: You have 7 at home! What do you do with all this stuff? Apart of cycling of course.\nSW: They mainly live in the garage, that\u0026rsquo;s why I had to move to Kent, so I can have a garage that can fit enough bikes in. Honesty I probably use one bike more than every other bike. I have my cargo bike which is like the big long one with a plank on the back, so I can carry several bags, several PCs, a crate a beer.\nKN: You\u0026rsquo;re doing this for shopping?\nSW: Yeah, you know. so I don\u0026rsquo;t drive. You can see the London ethos, I\u0026rsquo;ve never needed the car, because, you know, London. So if I\u0026rsquo;m going to the supermarket, I ride my big old bike and I fill up the back. It\u0026rsquo;s basically a pickup truck in bike form. That\u0026rsquo;s the way forward and it\u0026rsquo;s not too slow. I rode from Lyon to Marseille this year, it\u0026rsquo;s 400 kilometers on a cargo bike, it\u0026rsquo;s a long way to go. It was last year. It wasn\u0026rsquo;t last week.\nKN: Big challenge.\nSW: Yeah and I\u0026rsquo;m not fit enough for that these days. I\u0026rsquo;m generally unfit, but I\u0026rsquo;ve got good cycling legs.\nKN: Tell me about that journey.\nSW: The journey was a charity ride. So with Adatis we do sort of every year, every other year we do some big challenge thing, so we did London to Paris originally. It was good, rode down, got the ferry across, rode through. Last year was Lyon to Marseille, down the river, through Avignon, through Valence, through some beautiful places. It was good, it was warm. It was like middle of summer, so going up to Marseille, we didn\u0026rsquo;t quite realize there\u0026rsquo;s a massive hill. Like a huge hill going into Marseille and it was like maybe 30-32 degrees on a cargo bike. If there was a bus, I could have just hitched a ride with my bike, I would have been very happy. But we made it, it was good, didn\u0026rsquo;t break down, no major injuries.\nKN: Remind me how long the distance was?\nSW: 400 kilometers. And we raised a lot of money. So we raised over 10k for different charities.\nKN: So every year you\u0026rsquo;re doing this charity?\nSW: So far it\u0026rsquo;s every other year, but we want to start doing every year. So we\u0026rsquo;re talking about what we do this summer.\nKN: So every other year there\u0026rsquo;s a different activity?\nSW: Yeah. So we\u0026rsquo;re getting some pushback now and everyone\u0026rsquo;s like \u0026lsquo;Oh, did I have to be cycling again?\u0026rsquo; Obviously, I\u0026rsquo;m biased, I wanna cycle. We\u0026rsquo;ll see.\nKN: We are sitting in the building, what is it called, Code Node?\nSW: Code Node, yeah.\nKN: Tell me something about that user group. It\u0026rsquo;s London SQL or SQL London?\nSW: It\u0026rsquo;s the London Pass Chapter technically. So it gets called the London SQL user group, the SQL London user group, the many different user group things. I have a secret campaign, because just like every other SQL group in the entirety of the world, it shouldn\u0026rsquo;t be called SQL anymore, because we\u0026rsquo;re doing Data Factory, we\u0026rsquo;re doing all this other stuff, we\u0026rsquo;re doing Azure.\nKN: A lot of conferences changed their names.\nSW: Yeah, so the other group that I look after, SQL Surrey, is now the Surrey Data Platform group and something similar is going to happen to London at some point, but don\u0026rsquo;t know when yet. We\u0026rsquo;ll have a rebranding event at some point.\nKN: When will you reveal that?\nSW: When I convince the other organizers that we should do that. It\u0026rsquo;s by no means agreed, that it\u0026rsquo;s not an announcement, but it should come soon.\nKN: So another question related a little bit to technology, but we\u0026rsquo;ll go back to this. What is you hobby basically?\nSW: My hobby, oh. Okay, there\u0026rsquo;s a couple\u0026hellip; Cycling if you didn\u0026rsquo;t get that. I do a lot of board gaming, so I\u0026rsquo;m a massive table board gamer. The wife is a massive board game fan. We have a bookcase just full of different games. So anytime I can get a crowd of people around the table to play a game, I\u0026rsquo;m a big fan. Don\u0026rsquo;t talk to me about Monopoly, Cluedo it\u0026rsquo;s not that kind of thing. And cooking. I am an avid chef, I spent a lot of my time cooking. If I\u0026rsquo;ve got spare time I\u0026rsquo;m probably in the kitchen. And so my wife is French and she\u0026rsquo;s a big baking fan, so it\u0026rsquo;s good. So I make food, she makes cakes. She\u0026rsquo;s French, so patisserie and all that kind of stuff. So together we make people fat essentially. We make people fat, and then we get to play games and then I go for a ride to burn it off. And it\u0026rsquo;s all good. Kind of works.\nKN: Everything is balancing.\nWhen did you start your adventure with Microsoft Data Platform or whatever at the beginning?\nSW: 2005-2006 around then. So I started off actually 2004, I worked for IBM for a year and I had the horror of building reporting systems in Lotus Notes and all kind of stuff like that. That was a bad experience. Bit of Hyperion Brio, a few other things around that time. First job - load of Access databases, good old Access, building reports. We had this process on a Monday morning of this Access database, would churn through and create all of the company\u0026rsquo;s reports. And it would take like six hours using pure Access reports crunching and churning and doing all that stuff. And that was my first realization of how bad BI can be.\nKN: You heard during the university probably?\nSW: No, I didn\u0026rsquo;t do a tech degree. I did a business degree, I did nothing but management and accounting and marketing and economics and all of that kind of fluffy business-y stuff. I always knew I was always a techie. I spent my childhood building PCs with my dad in the garage, doing that kind of thing.\nKN: Yeah, I believe the passion is even more important than the official education.\nSW: I\u0026rsquo;ve always thought \u0026lsquo;Yeah, you can pick up tech\u0026rsquo;, but I didn\u0026rsquo;t know much about how business runs, how people deal with data, how people deal with marketing. And BI it\u0026rsquo;s good having that backing. Being able to go to a marketing department and say \u0026lsquo;What are your challenges?\u0026rsquo; And actually understand how they work in the theory and the fundamentals behind it. It kind of helps with that working with these different people going \u0026lsquo;I know a bit of HR, a bit of marketing, bit of accounting. Obviously we\u0026rsquo;re talking 15 years ago, so probably a bit rusty. I couldn\u0026rsquo;t walk into a marketing department and work as a marketeer, but it just helps having that familiarity. It gets you started in that conversation knowing what kind of challenges they face. And then yeah, we got SQL Server, we replaced all Access databases.\nKN: In the same company?\nSW: Oh yeah, we eventually, you know, we brought SSIS in, so I helped build the very first cube there. And a lot of it was kind of that internal\u0026hellip; didn\u0026rsquo;t really see a lot of the world, didn\u0026rsquo;t go to other conferences, kind of had that little bubble, we only knew what we knew and everything was an incremental improvement. I stayed there for seven years in the company. And then when I left, I joined Adatis, where I\u0026rsquo;m now. Three main jobs. And it was like going out into the world and going \u0026lsquo;right, all this stuff that I\u0026rsquo;ve learned and taught myself and worked with over 7 years, is that actually the way people do things, does that actually work?\u0026rsquo; And the answer is: some. There\u0026rsquo;s definitely things I didn\u0026rsquo;t know that well. Definitely things I\u0026rsquo;ve done that no one else had done. So a weird mix of\u0026hellip; a balance of skills and you get a lot of that. When people join the company we go \u0026lsquo;What do you know?\u0026rsquo; And it\u0026rsquo;s like \u0026lsquo;I know a little bit of that, a little bit of that, and tons about that!\u0026rsquo; And you kind of get that imbalance and that\u0026rsquo;s what\u0026rsquo;s there. And it\u0026rsquo;s about smoothing that out. Filling in any gaps, making sure people have that kind of a decent bit of knowledge about everything and then, yeah, specialisms when they know more than everyone else. That makes for a really good problem-solving team. You get a lot of people around the problem and there\u0026rsquo;s people who they all understand the same language that they\u0026rsquo;re are talking. But people have different depths and different viewpoints and different perspectives. If all the problems are fixed by one person, you got a very skewed solution at the end of the day. This is why it\u0026rsquo;s really funny at the moment, because this is whole shift. We\u0026rsquo;ve got like a load of the old guys. You\u0026rsquo;ve got the people who\u0026rsquo;ve been doing BI for 15-20 years and they have very firm ideas about how things work and they\u0026rsquo;ve got a massive depth of experience about a load of tools that people don\u0026rsquo;t use anymore. I\u0026rsquo;m really depressed, my MDX knowledge is just useless these days. And then you\u0026rsquo;ve got this whole group of youngsters coming in, you\u0026rsquo;ve got grad students coming in, who\u0026hellip; they\u0026rsquo;ve learned Python, they\u0026rsquo;ve learned R, they\u0026rsquo;ve learned some of the languages that we haven\u0026rsquo;t historically said \u0026lsquo;That\u0026rsquo;s a BI language\u0026rsquo;. It\u0026rsquo;s not C#, it\u0026rsquo;s not SQL, that\u0026rsquo;s not Microsoft BI. And now it\u0026rsquo;s core to everything that we do. The old guard were heavily just trying to learn a language that all our grads are like \u0026lsquo;Yeah, we know that\u0026rsquo;. So it\u0026rsquo;s good and it\u0026rsquo;s a nice challenge that we\u0026rsquo;ve got people\u0026hellip; the old guard can teach them \u0026lsquo;This is the way of working, the methodology, this is Kimball and this is the problem that you get if you build it in this way\u0026rsquo;. And then the other people coming in with a wave of new learnings and new ideas. You need that mix of experiences.\nKN: You have those experienced people in Adatis. Tell me about the team, the company.\nSW: So we\u0026rsquo;re now, I never know, 60-70 people, something like that. It\u0026rsquo;s different every time you ask. I joined nearly 6 years ago now and we were 10 people. I was like person number 10 and now to be 60-70 people, to go to the office and have people I\u0026rsquo;ve barely actually spoken to, it\u0026rsquo;s just a weird experience. But again, it\u0026rsquo;s really good, because we\u0026rsquo;ve now got that such a rich mix of background and where people are coming from, it\u0026rsquo;s good. So the culture is generally good, the culture is core to what we do. And it\u0026rsquo;s hard, you know, consultancy is hard. You have to travel to clients, you spend a lot of time on the road. And you have to have people expect you to be an expert on everything. They can be hammering questions, saying \u0026lsquo;How does that work, what do you do to that, what\u0026rsquo;s that scenario?\u0026rsquo; So it is a fairly pressurized job, but it\u0026rsquo;s also fun. It\u0026rsquo;s not for everyone, but I love it. That is what I do. I love consulting around the Data Platform, especially given how much it\u0026rsquo;s changing. If I was still answering the same old SSIS questions I was dealing with 10 years ago, then it wouldn\u0026rsquo;t be for me. But given that almost every 6 months the answers to the questions are different these days, because everything is evolving so quickly and everything is changing. And it\u0026rsquo;s been a difficult period for a lot of people, because it\u0026rsquo;s in so much flux. You started the project a year ago, you\u0026rsquo;d using Data Lake Analytics for everything. And now if you start a project, you\u0026rsquo;d be using Databricks for everything, and that\u0026rsquo;s caused a lot of uncertainty in people. People are saying \u0026lsquo;That\u0026rsquo;s what you\u0026rsquo;re telling me now, am I gonna believe you in 6 months\u0026rsquo; time, you\u0026rsquo;re gonna tell me something else\u0026rsquo;. So it\u0026rsquo;s difficult, but it\u0026rsquo;s fun.\nKN: What do you like the most in your role in the consultancy company?\nSW: For me it\u0026rsquo;s getting to work with a big range of people. Because of my role I don\u0026rsquo;t tend to be\u0026hellip; It\u0026rsquo;s both good and frustrating. You\u0026rsquo;ve got the really good feeling when you finish a project, you deliver it and you\u0026rsquo;re like \u0026lsquo;I built that\u0026rsquo;. But my role\u0026rsquo;s different. I don\u0026rsquo;t get that feeling of start to finish, because I get involved in lots of different projects and usually quite near the start, usually when there\u0026rsquo;s lots of technical uncertainty, I\u0026rsquo;m there to try and de-risk projects to say \u0026lsquo;Oh, we don\u0026rsquo;t know how to do that. Cool, this is how you would do that, this is the patterns, this is where Microsoft\u0026rsquo;s going. Being an MVP I\u0026rsquo;ve obviously got that kind of view into the product teams, into how things are working, where the road map is, being the gold partner, I\u0026rsquo;ve got a different avenue into the same, so I\u0026rsquo;ve got quite a good view of where things are going. So I can go in and give that recommendation and make sure people are designing it right. Setting up for success to use a horrible business term. But I get involved in lots of that early phase projects and then dragged in, you know, to do any technical spikes and technical troubleshooting. Now I never get that day of just going in and like \u0026lsquo;Ah, today\u0026rsquo;s gonna be easy. I\u0026rsquo;ve got just stuff I need to crack through\u0026rsquo;. My day on-site where the client tends to be \u0026lsquo;Ah, Simon, we\u0026rsquo;ve saved up all these horrible problems that we don\u0026rsquo;t know how that works\u0026rsquo;. I tend to have fairly stressful days on-site, but it\u0026rsquo;s good. It keep it interesting, it\u0026rsquo;s always new problems that haven\u0026rsquo;t been solved yet. Because if it\u0026rsquo;s already been solved, then we\u0026rsquo;ve shared the answer and people can go and get the answer. So it\u0026rsquo;s an interesting role, it\u0026rsquo;s tough, but it\u0026rsquo;s exciting. Again, I get bored easily. So you know, having a role that changes\u0026hellip;\nKN: Problems that are easy to resolve are not very satisfying.\nSW: If we\u0026rsquo;ve fixed it before, I\u0026rsquo;m not interested.\nKN: So having all these changes every month, every week and all this technology that you\u0026rsquo;re working with. How you are trying to keep yourself up to date?\nSW: It\u0026rsquo;s not one man. So we\u0026rsquo;re very keen on R\u0026amp;D, so we have a whole research and development process where all of our consultants are expected to be researching something new. So it\u0026rsquo;s not just \u0026lsquo;Ah, you\u0026rsquo;re on a boring project, it\u0026rsquo;s not gonna be interesting\u0026rsquo;. Part of our progression, part of the skills path, the career path is about doing R\u0026amp;D, about blogging, about finding a subject that\u0026rsquo;s your own. So I might be sitting there going: \u0026lsquo;Okay guys, there\u0026rsquo;s a new thing coming out. Can someone go and look into it and write it up and write a blog and compare performance between these things? Or write a pattern for have to do that\u0026rsquo;. That\u0026rsquo;s not me doing all of that stuff, that\u0026rsquo;s impossible for one man to be on top of all. I take the interesting ones for myself. I\u0026rsquo;m gonna make sure that people are doing it. I spent a lot of time reading the blogs and reading the output that our guys have done so that I can get good advice, but\u0026hellip;\nKN: At least you need to read a lot. But it consumes time.\nSW: I spend a lot of time on trains.\nKN: So you have this time. This might be a very hard question, depends on you. Which achievement are you satisfied with the most?\nSW: I don\u0026rsquo;t know. So the MVP means a lot. That kind of thing of working in the community for a long time and kind of doing talks and running user groups, that kind of recognition means a lot, it is good. I don\u0026rsquo;t know\u0026hellip; Lots of achievements. I don\u0026rsquo;t really keep track of that. I come across as a fairly egotistical, but actually there\u0026rsquo;s not many things that I do just for me. Most of the time it tends to be a team\u0026rsquo;s done things, that kind of stuff. There\u0026rsquo;s plenty of solutions that we\u0026rsquo;ve built, that were designed, and I\u0026rsquo;m like \u0026lsquo;Yeah, that\u0026rsquo;s really cool\u0026rsquo;. Yeah, I really like some of the stuff that we did. The data science and stuff. And that doesn\u0026rsquo;t come naturally to me. I\u0026rsquo;m not a trained data scientist. I used to do stats a lot, but not a lot these days.\nKN: So the really last question at the end of our conversation. Tell us where we can find you.\nSW: All over the place. If there\u0026rsquo;s a conference happening and it has to do with Azure and data I\u0026rsquo;m probably there recently. So big things, got a pre-con coming up in SQLBits. So myself and Terry McCann doing a Databricks deep dive. It\u0026rsquo;ll be fun, cause we\u0026rsquo;re doing data engineering versus data science. So he\u0026rsquo;s repping the data science side of thing. I\u0026rsquo;m doing the real work. Also doing intelligent cloud conference in Copenhagen. Doing another pre-con around modern data warehousing, so covering all the different facets of that. I\u0026rsquo;m on Twitter, I haven\u0026rsquo;t blogged in an age. I get people to blog for me. I keep people to do R\u0026amp;D on certain subjects, so I wouldn\u0026rsquo;t take credit for their work obviously. But yeah, @MrSiWhiteley, it\u0026rsquo;s me on Twitter. Generally talking about Azure-y things and if anyone has any questions about how BI works in Azure, how data science fits in, how you do any of this stuff, what\u0026rsquo;s the point in a Data Lake, that kind of question I just get asked on a daily basis. So if anyone wants to know anything more about that, feel free to reach out!\nKN: Thank you for this conversation today.\nUseful links Simon\u0026rsquo;s Twitter: @MrSiWhiteley\nSimon\u0026rsquo;s Post activities on Adatis Blog\nSimon on LinkedIn\nWe were talking about Azure Data Factory, Azure SQL Datawarehouse, SQL Server Integration Services (SSIS), Azure Databricks.\nSimon is one of the leaders of the PASS SQL London User Group\nExciting News! We've wrangled some extra seats for our previously sold out precon @SQLBits in 2 weeks! If learning all things #Azure #databricks with @SQLShark and I is your bag, sign up at https://t.co/fM1eVtGuP3! I made a little teaser for you guys... pic.twitter.com/I2qN8rdW12\n— Simon Whiteley (@MrSiWhiteley) 11 February 2019\r","date":"2019-02-22T07:00:58Z","image":"/uploads/2019/02/ASF019-PASS-SQL-London-with-SimonWhiteley.jpg","permalink":"/2019/02/asf-019-simon-whiteley-interview/","title":"ASF 019: Simon Whiteley interview"},{"content":"I\u0026rsquo;m super excited about the fact I\u0026rsquo;ll be speaking at SQLBits this year. It\u0026rsquo;s my first time as a speaker at that fantastic event and 3rd time as a volunteer. Looking forward to it! As always, it will be grueling and very well spent time among other fantastic speakers, friends and #sqlfamily members in general.\nSQLBits is coming up so fast - the event will take place between 27th February and 2nd March 2019 in Manchester. Don\u0026rsquo;t you have a budget? It\u0026rsquo;s not a problem - Saturday is completely free, so reject all excuses and register now.\nSaturday, 1:45 pm. See you? I\u0026rsquo;m speaking on conferences since 2014, but this, as I mentioned, will be my first time on SQLBits. What topic do I cover during my session? For 50 minutes, I will talk about the new feature in Azure Data Factory: Mapping Data Flow. The feature is still in a private preview stage, so you might have not seen it yet, but I can assure you that you would be interesting those capabilities which may replace SSIS in the future.\nAzure Data Factory v2 with (Mapping) Data Flows Capabilities Microsoft\u0026rsquo;s services in Azure helps us to leverage big data more easily and even more often accessible for non-technical users. Having UI in ADF version 2 - Microsoft added a new feature: Data Flow which resembles components of SSIS. This is a very user-friendly and non-code approach tool-set. But, has that been only UI introduction? Why and how Databricks does work under the hood? Do you want to know this new (still in private preview) feature of ADF and reveal the power of modern big data processes without knowledge of such languages like Python or Scala? We will review this new feature of ADFv2, do deep dive to understand the mentioned techniques, compare them to SSIS and/or T-SQL and learn how modelled data flow runs Scala behind the scenes.\nI commence my session on Saturday @ 1:45 pm in room 1 (called \u0026ldquo;Cagey Joe\u0026rdquo;).\nPlease come and say hi. You\u0026rsquo;re more than welcome!\nPS. If you still not sure whether to come or not - take a look at these videos or check that post out.\nCheers,\nKamil\n","date":"2019-02-18T19:05:19Z","image":"/uploads/2019/02/SQLBits-logo-speaking.gif","permalink":"/2019/02/speaking-at-sqlbits-in-march-2019/","title":"Speaking at SQLBits in March 2019"},{"content":"Press Power BI Desktop February 2019 Feature Summary\nPreview of new Key Influencers visual and much more\u0026hellip;\nThe February release of Azure Data Studio is now available\nExtension packs are the coolest feature.\nGet started quickly using templates in Azure Data Factory\nThis is a big step forward in ADF. It also applies to (Mapping) Data Flow.\nProtecting SQL Server Data Using Static Data Masking\nGreg Larsen (T) explains various data masking methods and presents how to mask a database with SSMS.\nHow fast can a $5,436/mo Azure SQL DB Hyperscale load data?\n\u0026ldquo;it’s like they discovered a new species, not like they wrote the code themselves\u0026rdquo; - Brent (T) said.\nPower Query Source (Preview)\nThis new connection has appeared in SSIS which can be used in ADF via IR (Integration Runtime). I\u0026rsquo;m waiting for this in ADF.\nDeeper integration and new connector for SSIS in ADF\nThe step-by-step manual shows how to leverage the Power Query in ADF.\nA word from Microsoft’s CEO Microsoft has always been a partner-led company, and we are committed to creating more opportunity for our partners across our businesses. https://t.co/kV8WPNEOAj\n— Satya Nadella (@satyanadella) 5 February 2019 Video: Azure Cognitive Services improves citizen engagement https://www.youtube.com/watch?v=mIGefGYNW3s\nSmile Corner ","date":"2019-02-17T21:26:27Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/02/last-week-reading-2019-02-17/","title":"Last Week Reading (2019-02-17)"},{"content":"Check how many bicycles he has got at home and where was the destination of last year of a 400 km trip?\nWhich tool among Azure is the closest to his soul? What exactly is hiding behind the term \u0026ldquo;Modern Data Warehouse\u0026rdquo;?\nWhen Azure Data Factory (Mapping) Data Flow will be publicly available?\nAlso, we will try to answer the fundamental question: is the SSIS dead or alive?\nNew podcast\u0026rsquo;s episode is coming (next Friday) and in the meantime, take a look you can watch that episode\u0026rsquo;s trailer.\nSimply click the PLAY button above or move to the YouTube: https://youtu.be/L2Xs9z9TDBY\nCould you guess who is the next guest?\nEnjoy!\n","date":"2019-02-15T07:00:07Z","image":"/uploads/2019/02/ASF019-video-invitation.jpg","permalink":"/2019/02/asf-019-video-invitation/","title":"ASF 019 video invitation"},{"content":"Hello Data Folks! It has been a very fast week and an extensive weekend whilst I have been at SQL Saturday in Krakow, Poland. I did not submit any subject this time, so I turned on a relax-attendee-mode meeting with all friends and learning new stuff over there. More about it next time - now let\u0026rsquo;s have a look at what the data world brought us last week.\nPress Two new Azure analytics GA releases, one preview and a big push on SQL DW\nNot really true about ADF in public preview but fine with rest of the news.\nIndividually great, collectively unmatched: Announcing updates to 3 great Azure Data Services\nThree updates about services: Azure Data Lake Storage (ADLS), Azure Data Explorer (ADX), Azure Data Factory (ADF) - Mapping Data Flow.\nNew connectors added to Azure Data Factory empowering richer insights\nADF supports 80+ source connections. I told you they adding connection every month. Check these enhancements out.\nAnalyst Report: Data Warehouse in the Cloud Benchmark\nAzure SQL DW vs. AWS Redshift, Snowflake, Google BigQuery benchmark. SQL DW is 2x faster than Redshift, 7x faster than Snowflake, 14x faster than BigQuery (and 17x cheaper too).\nAzure DevOps Projects supporting Azure Cosmos DB and Azure Functions\nCI/CD pipelines are available for these services.\nAzure Stream Analytics now supports Azure SQL Database as reference data input\nAnother input (the most demand for users voted on UserVoice) for Stream Analytics is in public preview now.\nDownload and install sqlpackage\nA new release of sqlpackage is now available on all platforms! This includes support for UTF8 collations, fixed significant schema compare performance issue, and moved to .NET Core 2.2.\nSQL Server Community Slack - a new channel\nWith Chrissy LeMaire (T) help, I created a new public channel: #azure-data-factory. Come over and discuss.\nSQL Saturday #824 Krakow\nI have been at this event last week, meeting my polish and foreign friends. Presentations are ready to download.\nVideo: China\u0026rsquo;s rise in artificial intelligence https://www.youtube.com/watch?v=6yDw9w8uPyQ\nSmile Corner This is brilliant, hence I decided to put this video here. Do you have these skills? Hit production! ;)\nhttps://m.youtube.com/watch?v=0vPt7GI-2kc\n","date":"2019-02-11T09:58:25Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/02/last-week-reading-2019-02-10/","title":"Last Week Reading (2019-02-10)"},{"content":"Press Webinar Follow-Up: Top 4 Data Technology Trends for DBAs in 2019\nVery interesting discussion panel led by Kevin Kline from SentryOne with his guests: Mark Allison, David Klee, and Chrissy LeMaire)\nAnnouncing the general availability of Query Store for Azure SQL Data Warehouse\nJust turn it on for your Azure SQL DWH.\nFree Downloads for Powerful SQL Server Management\nBrent Ozar gathered the most popular scripts \u0026amp; tools in one place.\nPowerShell export Databricks workspace items - recurse\nIf you want to build your CI/CD for Databricks Notebook - recursively export all items with PowerShell module.\nCloning Databases with PSDatabaseClone\nSander Stad will show that you can clone a database without any additional tools.\nFAQs About Organizing a Data Lake\nMelissa Coates answers for common questions about how to organise the files in your Storage Account.\nAsk SQL Family - SQL Player\u0026rsquo;s show on Stitcher\nI\u0026rsquo;m happy to announce that \u0026ldquo;Ask SQL Family\u0026rdquo; podcast is available on Stitcher platform since this week. Enjoy listening!\nWelcome to Michal Sadowski\nAgain, I\u0026rsquo;d like to warmly welcome to the new member of SQLPlayer blog: Michal Sadowski. Michal, good luck with all your plans for this year!\nA word from Microsoft’s CEO With our Power Platform, we are enabling anyone with a big idea and the passion to make a difference to maximize their impact and transform their organization for the better. https://t.co/piRUeB1QP8\n— Satya Nadella (@satyanadella) 30 January 2019 Video: Bringing AI to the edge: Azure Cognitive Services in Containers https://www.youtube.com/watch?v=hdfbn4Q8jbo\nSmile Corner ","date":"2019-02-03T21:09:11Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/02/last-week-reading-2019-02-03/","title":"Last Week Reading (2019-02-03)"},{"content":"Hello guys, a few internet posts that you might find interested.\nPress Azure SQL Analytics—Automating Deployment\nApparently, it wasn\u0026rsquo;t an easy piece of work as Joey D\u0026rsquo;Antoni has thought.\nGenerating Reports Using Word Documents And Flow\nYou can test that trick because Microsoft Word template has been released and Chris Webb describes how.\nUsing AI and IoT for disaster management\nAI and IoT are helping to tackle global climate change.\nHow to Copy Multiple Tables in Azure Data Factory\nSimple and very effective way.\nRun Azure Functions from Azure Data Factory pipelines\nThis Azure Friday video shows how to run a Function from ADF pipeline.\nOn-premises data gateway management in the Power platform admin center\nNow, this feature is in Public Preview.\nDAX Studio\nMany new features and bug fixes in the new version of DAX Studio 2.8.1.\nTop 14 MOST famous Python libraries \u0026amp; frameworks\nI was interested in to find the most popular ones.\nA word from Microsoft’s CEO Today we celebrate the incredible life and legacy of Dr. Martin Luther King Jr. I hope Dr. King’s work inspires each of us to empower others and to stand for change in our own communities and society #MLKDay https://t.co/5PF9aPDTLw pic.twitter.com/uEEgvRrUaq\n— Satya Nadella (@satyanadella) 21 January 2019\rVideo: Drones vs. California\u0026rsquo;s wildfires: How they\u0026rsquo;re helping firefighters Smile corner ","date":"2019-01-27T23:20:34Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/01/last-week-reading-2019-01-27/","title":"Last Week Reading (2019-01-27)"},{"content":" Introduction Mladen Prajdić is a Data Platform MVP from Slovenia. He\u0026rsquo;s been professionally programming for over 15 years, developing different types of applications in .Net (C#) and SQL Server, ranging from standard line-of-business, image-processing applications to high performance and IoT applications. He\u0026rsquo;s a regular speaker at various conferences and user-group meetings, really likes to optimize slow SQL statements, analyze performance, and find unconventional solutions to difficult SQL Server problems. In his free time, he also develops a very popular add-in for SSMS, called the SSMS Tools Pack (www.ssmstoolspack.com).\nThis talk has taken place during SQL Saturday #782 Slovenia conference in Ljubljana, on 8th December 2018 (Saturday).\nInterviewers: Kamil Nowinski \u0026amp; Michal Sadowski.\nHow many people do live in our guest\u0026rsquo;s country? What can give you a valuable perspective? How can you convince your boss to upgrade SQL Server to a newer/latest version on enterprise platform? Find out the answers to these questions and much, much more.\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy! [/paragraph_right][/row]\nTranscript Kamil Nowinski: Hello, Mladen. Thank you for accepting my invitation for this podcast.\nMladen Prajdić: You’re welcome, glad to do it.\nKN: At the beginning, could you tell me what your name is and where you live?\nMP: So my name is Mladen Prajdić and I live in Ljubljana in Slovenia. It’s a very small country south of the European Alps. It’s a beautiful country.\nKN: How many citizens are there?\nMP: In Slovenia? There are 2 million people in Slovenia.\nKN: And in Ljubljana?\nMP: Ljubljana has about 250,000-ish. Depending on what you count as Ljubljana. But it’s a very small country, but you’ve been here more than once, so you know about our hospitality and how the nature is beautiful and how the people are friendly. We try to treat you right.\nKN: Yes, and basically the city centre looks amazing during December and winter.\nMP: Yeah, with all the Christmas lights and everything. It’s pretty romantic.\nMichal Sadowski: What are you doing for a living?\nMP: So basically, I’m a developer. Right now, I’m developing for iOS. I’m doing Swift development, but I basically specialize in databases and database development, C#, stuff like that. I like to do performance tuning as well, on both sides of the IO, on database side and the application side. I do sessions about that as well and I organize the SQLSaturday Slovenia. Well, I help organize, I’m one of the organizers. I go around and give sessions around the world. Different SQLSaturdays and other conferences. So yeah, it’s a fun life.\nMS: Could you tell us a little bit more about the tool you have created, this SSMS Tools Pack? When was the first release?\nMP: The SSMS TooIs Pack is basically an add-in for SSMS. I created it in 2006, that was the first version.\nKN: At the beginning it was free version, I think.\nMP: So the story was, I was talking on about 27 December 2005, I was talking to a friend in the States and he said “You know, it would be really nice to be able to somehow add functionality to management studio” and I’m like “Challenge accepted”. So it basically took two weekends and reverse-engineered the steps to extend SSMS, and I wrote what he wanted, basically a CRUD generator, CRUD stored procedure generator. That was the first feature. So I created that, gave him the test version, he was like “Oh my God, this is amazing”. And then I said “OK, this might be useful to other people as well”. So I started developing more and more features and the first official public version was about six months later. I can’t remember what features there were at that time, but since then it has grown in popularity a lot. And it’s free for SSMS 2008 and SSMS 2008 R2, it’s still free. 2012 and upwards requires licensing, because in 2012 I said “Well, this is consuming so much of my free time”, because this is a project that I do in my free time still. It consumes so many resources and my free time that I either open source it or start charging for it. And I decided against open sourcing it, because it was something that Microsoft hasn’t supported officially. It still isn’t supported officially. And I would still be the only one working on it to be honest at that time. So I decided to start charging for it and do proper customer support and everything. And then it just grew, and now it’s grown a lot.\nKN: Can you say how many customers you have for this or how many licenses you’ve sold?\nMP: Well, no. I mean, I know, but I won’t tell you. But I can tell you that since I released it in 2005, I had about 1.3 million downloads since 2005. So this is all versions and everything.\nKN: When you started working on that, did you have any documentation how to create add-ins to the SSMS or did you ask Microsoft for help?\nMP: No, there was no documentation at the time, there still isn’t. I basically took Reflector, so that wasn’t Red Gate’s tool at the time, it was still open source I think. Somebody created it, I don’t know, I don’t remember who, but it basically showed .NET DLL’s in disassembled form. So I took that and just went through how SSMS works. I reverse-engineered it and I used Sysinternals tools to see what folders SSMS reads at the startup to see where you should put some files that are recognized by the add-in model. Because SSMS was built on top of Visual Studio, which supported add-ins at that time, so it would make logical sense that SSMS also does something similar. It was just completely unsupported by Microsoft. And to this day, every new version of SSMS that comes out, it breaks something from the previous version, so I have to rewrite some infrastructure code.\nMS: Do you still use Sysinternals tools for that? Just to reverse some of the procedure, or do you have any other tools?\nMP: No, I still use the Process Monitor and Process Explorer to see where it goes. It’s still the basic tools for this task. And of course now I use ILSpy, this is an open source disassembler. And that’s pretty much it. And it’s still the same process: going, see what happens, reverse engineer, try it, if it works, it works, if it doesn’t, it doesn’t.\nKN: Next question: about the MVP. What do you think about the MVP those days? MVP as a program?\nMP: I have no issues with it, to be perfectly honest. I mean, sure, things could be better, but things could be always better. As a group of people, we have some talking about MVPs in general. Everybody has their own views how something should be run, but in the end it’s Microsoft’s program. Microsoft decides how it should be run. I mean, we can either go with it, go against it, or try to improve it. And I think that most people are trying to improve it in their own way. I mean, sure, things could be better, but I have no real issues with it. I mean, it gives you great benefits, because you get\u0026hellip; I don’t know, what I can say that’s not covered by the NDA. You get some access to people that you wouldn’t get access to otherwise.\nKN: My next question would be like what specific benefits are important or useful for you?\nMP: For me basically meeting people. Through the MVP program, MVP Summit, there are different communication channels, stuff like that, gives you the opportunity to meet people that you might not get to meet otherwise. Because they’re highly technical, highly knowledgeable and, in the end, it basically comes down to that if you have an issue that you cannot solve, you know who to ask that can point you in the right direction of the solution. And, in my opinion, that’s the most important thing that you can get. Because you can buy licenses, you can google stuff, but you have to know what to google for. Once you know what to google for, it’s easy. You have to know what to google for.\nKN: And also you can get the answer much, much quicker and directly from the source. Last question about the MVP. When were you nominated, when were you awarded for the first time?\nMP: 2010. I’ve been an MVP for a while. 8 years now. Time flies.\nMS: What hints would you give to young people who want to start working on the market, I mean the IT market. In general.\nMP: It’s not as simple as it sounds, the IT market. You have to be prepared to learn constantly. I mean technology these days is going so fast, it’s developing so fast that you just can’t keep up. Realistically expecting that you can keep up with everything is impossible, so you have to find one or two things, preferably two things that you can be really good at, that you like doing, and just go do that. There is enough work in those two areas, no matter what the areas are. There’s more than enough work for you and you can always switch areas if you want. For me that was C# development and T-SQL development. I don’t do DBA. I do DBA but I don’t like doing it. Also one important thing I think that they should know, is learn the basics. So I see a lot of young people just jumping into the waters of IT without knowing basic stuff. Like if you’re a developer, you don’t know basic algorithms for things, or basic data structures. And sure, you might think that’s not important, but in the end it is important, because it gives you perspective on how you can solve different business problems. Because what is programming in the end? It’s translating business problems into solutions via code, and that’s just algorithms in a different way. For other kinds of IT, since I don’t specialize in those, I don’t have any other advice, but you have to know the basics, you have to have a good foundation that you can build on.\nMS: Then how can they reach that kind of foundation? How can they learn those things?\nMP: There’s a lot of resources online. Which one you choose is completely up to you, but once you start reading into something or listening to podcasts, or learning from videos from different websites or whatever the method is, you will see which ones are good. Because not all are created equal. But you don’t know which are good and which are not, because you don’t have enough knowledge to decide which are good and which are bad. So once you get to know some stuff about the subject you choose, then you can decide “OK, this is not giving me the things I need” and you can find something better. So in the end it doesn’t matter, just type into Google the subject that interests you and start reading on the first link. And then you will find either books that are good or videos that are good. I mean, if you just say “OK, I want to learn programming T-SQL” and you go grab a book “Beginning T-SQL Programming”, sure. I made up the title of the book, I don’t know if that exists actually. But you don’t know if that book is good or not. Sure, you will start reading it and then after 500 pages you will see “OK, that maybe is not that good”. You have no clue. When you start, it doesn’t really matter, where you start or how you start. You just have to start. And you have to be prepared to learn a lot.\nMS: Do you think that it’s required to have university background just to start working in IT? You need to graduate in computer science or something similar?\nMP: No, I don’t think so. Because I know a lot of people who are really good IT people programmer system analysts, system engineers that do not have official technical background. You might run into issues when you try to get, I don’t know, visas for some countries or something like that. That if you have, say, a social studies degree and you want to get a visa to, I don’t know, United States or whatever. And you say “OK, I’m gonna work in programming or in IT” and they will say “But you have a social studies degree. That does not work together. How are you gonna prove that you know what you’re talking about? Or do you have any certificates?” So from this perspective, I would say that getting a degree is very, very good. For me it potentially opened a lot of doors that I would have to work for much harder otherwise.\nKN: And what about certificates then?\nMP: So personally, I don’t do certificates. It’s not because I wouldn’t want to or I think they’re bad or anything, because they’re not. They can be very useful. I just don’t have the need for it. So it depends on what you want and the situation. If you go for an interview to a company that you can see that they like certificates, go to them, it’s fine. You will learn new stuff. That’s always great. But I don’t do certificates mostly because I haven’t had the need to do them yet.\nKN: So as you mentioned, we know that you are speaking a lot. Not only in Europe market. Also across the pond. And how do you prepare yourself for a speech?\nMP: For a session? I don’t actually do a lot of preparation because I’ve been speaking for more than ten years now, so I know my timings by head. I don’t have to go in front of the mirror and practice. I just ride the slides and it doesn’t matter what I write on the slide time wise. It doesn’t matter because even if I write just two words about the subject, I can still extend that to ten minutes if I have to. And if I don’t have the time, I can just compress it into one minute. So I know usually where I am time wise. As for content, I always try to overprepare. Because if, for example, I would now give a session on, say, high ability. I don’t know much about high ability, because I never really did a deep dive into HA before. If I had a session about that, I would do a complete deep dive into that and learn every single thing I can, because you can never be overprepared. You want to answer as many questions that people give you as possible, but never be afraid to say “I don’t know”, because nobody expects you to know everything. If somebody expects you to know everything, it’s illusion.\nKN: Sometimes people expect, especially if you are an MVP.\nMP: They can expect, but that’s not realistic. Other than that, I don’t do much preparation.\nMS: How do you start with the presentation? Do you start with a blank sheet of paper and pencil and then you just draw all of the items that you would like to discuss, or do you start with PowerPoint or some other tool?\nMP: If I were to build a completely new session, first I decide what I want to talk about. I write the title. Then I write the abstract, because the abstract gives you clarity on what you want to say. What you want to have the session about. And then I open PowerPoint and put about ten key points. One per slide, so I have ten slides for example and put ten key points in each slide, which I think are the most important for that session. And then I expand on each key point on one slide, but I don’t overdo the slides. And when I do that, I look at the session, I take a break of about a week and then probably completely rewrite it. But why? Because for me it takes about a week to put all of the pieces that I put in PowerPoint already to put them together in my mind and see “OK, the flow is maybe not that good, here it can change etc.”. And then I give the session and then I see a hundred thousand different things that I could change and then I change it again. So every session is different and every session, every presentation, is basically a work in progress.\nMS: Do you have any, let’s say, freeze code just before the presentation? That you are not changing anything in the presentation before the presentation? Or you are making the changes like two minutes before “Oh, I just need to change this slide” and then\u0026hellip;\nMP: No, I have a rule that I don’t change anything in my slides at least three days before the presentation. So I do a test run about two days before the session. I do a test run of all the demos and slides and everything. If I find something that I’ve made wrong or anything, I fix that. But generally, as for big changes, I don’t do that. And then I do another test run of demos about a few hours before the session.\nKN: Because Azure is changing every day.\nMP: That’s true. Luckily, I don’t do much Azure demos. I try to do as little as possible because of that reason.\nMS: How about work-life balance?\nMP: As you get older, that changes. When I was 25 I think, I said “I will not overwork myself”. Because I’ve done a lot of work before I was 25 and I said simply “No”. Now I have a very strict differentiation between work and life. I do eight hours at work and then, when I go home, I put work on hold and I deal with my pet projects like the SSMS Tools Pack or family or whatever. And basically I think family has priority always, but it’s easy for me to say as I am not a consultant or I’m not self-employed. I’m an employee in a company, so it’s probably easier for me to say that than for somebody who’s a consultant who has to take care of their own cash flow.\nMS: What is the personal achievement that you are most proud of?\nMP: Personal achievement? So when it comes to the technical side of things, it’s definitely sticking with the Tools Pack for so long. It’s been 12 years now. As for personal, I don’t know. I mean, I’ve done a lot of stuff, I’m proud of all of them.\nKN: Now everyone knows that we do not send our questions in advance.\nMP: These questions\u0026hellip; like, personal achievement, what the hell? It was the hardest question.\nKN: Which part of your life, of your work is the hardest?\nMP: That’s a really good question, very hard to answer. I would say the hardest part of life and work is getting well and along with other people. Because we are not all people persons, right? So when you work with people, you get a lot of different personalities and everything, and nicely navigating those borders – it can be quite hard. And same in personal life. So I think that’s the hardest part in everybody’s life. You know, getting along with other people. The communication thing, I think that’s the hardest point. Everything else is just a matter of doing it. If you don’t want to do something, you won’t do it.\nKN: What do you think is the future of on-premises servers?\nMP: I think it has a future. I’m not worried about them at all because there will always be customers that, for one reason or another, won’t be able to go into cloud. And the cloud is not a magic solution for everything. So I think they will stay here for quite a while.\nKN: How do you think we can help customers or companies migrate to the newer version of SQL Server? Either on-premise or to the cloud.\nMP: You have to show them the business value for it. That’s simply it. IT is a cost center, it’s not a profit center, so you have to have the skills to turn it into a profit center somehow. Or lessen the cost of it. Now, that’s a business value. And if you can show that, then you’re done. Most people in management positions think of IT as a cost center. So how do you position yourself as a profit center? You have to position yourself as a problem solver for the profit centers in the company. Now what are the profit centers in your company? That’s up to you to decide. I don’t know who the profit centers are in your company. Either its production or marketing or sales. And then you talk to those people and solve their problems. That’s pretty much it. And if you can show a company that their profits will go up because of moving to a newer version of something, then you should be able to sell that.\nKN: So we already know what is the main reason why companies are not emigrating to the newer version.\nMP: They don’t see the reason for it. I mean, the place where I work, we still have one 2000 server, because there’s no actual need to change it. Sure, it isn’t open to anywhere outside, it has one table that is used by five different applications, but that server is enough to run that table. It’s not a high speed table, so there is no need to buy another license for that server.\nKN: But it’s not supported. What if something happened to this database or maybe if you can’t maintain some application?\nMP: The database would be just migrated to another server. If one of those things where if things work, don’t break them or don’t fix them. Well, don’t break them, of course not, but if they work, don’t try to fix them. You have to be pragmatic about it as well.\nKN: How many customers do you think are ready to migrate to Azure?\nMP: Percentage-wise or\u0026hellip;? I have no idea.\nKN: It’s changing a lot. Azure is more popular year by year.\nMP: I think a lot of small businesses could easily move to Azure, because of all the infrastructure thing. And so they don’t have to keep their own IT.\nKN: It’s easier for startups.\nMP: Not start-ups, smaller customers, mom-and-pop places, smaller businesses, up to say 50 people. I think there are prime candidates for that. As to how are you going to convince them to do that? Show them the profit in it.\nMS: What do you think about the current trend of migration to cloud? Is it going to be replacing fully SQL Server that is on-premises, or will it be that the market will be divided into two separate silos that will remain the same. So like 80% in Azure and then 20% on-premises, or will it be changing?\nMP: Right now that trend is definitely going up. People are migrating to Azure. Now, we’ll see what happens in the future. If they are happy with the services they receive, sure, they will stay there. If not, they will move back to on-premises. But I think on-premises isn’t going anywhere. In the end it’s going to be a mixture of some kind and if you can build something that eases that distinction between on-premises and in the cloud that bridges that gap, that will solve a lot of issues. There won’t be either one or the other, there will be a mixture of both. What the percentages will be, I have no idea. That will totally depend on the success of cloud offerings and features.\nKN: The last question is how did you start working with SQL Server?\nMP: So, when I was a student, I started working in programming in 1999. And at that time I worked with Microsoft’s Visual C++ and Oracle. And after that, I swore I would never touch databases again. And I haven’t! Fast forward to 2002 I started working with .NET in a different company. My boss at the time comes up and says “So we have to do this Pivot thing, we have to change this data into grid form etc.”. And I don’t remember what he was explaining at that time, but I was like “OK, fine”. So I started programming things in C# and I’m like “There has to be an easier way”. So I’m like “How do you even start searching for this?” Because come on, this is 2002, Google was very non-existent at the time. I mean, they were there, but you know. So I found a website called sqlteam.com. It was forums, had a lot of articles and I asked a question there. They told me it’s called transposing data or pivoting data or something like that. And I said “OK, that sounds promising”, so I started looking into it and I wrote my first T-SQL statement to do a Pivot on that data. I wrote it, I don’t know, in two days and it magically worked and then I started asking and answering questions on forums on sqlteam.com. And then I started blogging and then I started speaking and then I started the Tools Pack. Well, I started the Tools Pack, then I started speaking. And now we’re here. So that’s how I started working in SQL.\nKN: In 2002. Sixteen years.\nMP: That’s a lot of time.\nKN: At the end of our conversation, tell us where we can find you. I mean on the Internet of course.\nMP: If you google me, I will be two pages of Google. Seriously though, I have a blog at sqlteam.com, which has been inactive for quite some time. You can visit me at ssmstoolspack.com, that’s the page of my add-in. You can find me on Twitter, so @MladenPrajdic. I’m on Facebook, but I’m not a lot on Facebook. You can mostly get me on Twitter.\nKN: OK, cool. Thank you very much for the conversation.\nMP: Thank you, it was fun.\nUseful links Mladen\u0026rsquo;s Twitter: @MladenPrajdic\nMladen\u0026rsquo;s LinkedIn Profile\nSSMS Tools Pack: official website\n","date":"2019-01-25T01:45:47Z","image":"/uploads/2018/12/ASF018-SqlSatSlovenia-with-Mladen-Prajdic.jpg","permalink":"/2019/01/asf-018-mladen-prajdic-interview/","title":"ASF 018: Mladen Prajdic interview"},{"content":"Before I publish the next episode of \u0026ldquo;Ask SQL Family\u0026rdquo; podcast, I would like to stop and look back for a while. Any time is good for that kind of summarize, but New Year especially.\nAs I mentioned in that post recently, the podcast is not the most popular category on this blog. Nevertheless, I thought that I might share some numbers which I didn\u0026rsquo;t follow particularly before.\nPodcast in numbers Hence, let\u0026rsquo;s reveal some numbers and statistics about the podcast:\n1,057 plays 925 downloads 17 listenable episodes published to date Almost 12 hours of recorded conversations 41'44\u0026quot; is an average length of all episodes 8 different places where the chats have taken place 2,796 visits on interview\u0026rsquo;s pages The chart below comes from **Spreaker**where my podcast is hosted. Unfortunately, I cannot download the statistics before 2018.\nWhen I came up with this idea? The main idea has existed in my head a far away before the blog started.\nIn 2017, when the SQLDay conference was approaching, I already knew that the blog must start soon. So, I thought that the conference (this and others) would be ideal places where I can conduct conversations with well-known, well-liked and valued specialists from the world of SQL Server.\nFrom the very beginning, it was known that the conversations will be recorded and then transcribed into articles. When I talked about it with my friend (greetings to Mateusz) - he immediately suggested that it would be a very good material for the podcast. At that time I was not sure what the quality of the recordings will be because I did not prepare myself for such a scenario. But - why not? I like challenges after all.\nThen I invited and asked Damian whether would he be interested in conducting the interview along with me as I have seen many positive aspects of such format. After the first two recordings during the SQLDay conference - it turned out that the quality is better than I expected and is suitable for publishing as a podcast. That\u0026rsquo;s how it was.\nThe conversations have been recorded, in the meantime, I have been working hard to prepare the start of the blog. Categories, layout, skin - everything mattered, and I do not like anyhow-quality.\nIssuing in the monthly cycle is quite a challenge. Either way, it still manages to keep up the pace and I hope it will stay this year as well. The guest list is prepared with a surplus.\nWho have we met? In this short video, you will see all my/our guests we have met so far.\nhttps://www.youtube.com/watch?v=gs0PhoN0Ni0\nOther podcasts To be honest - there are more very good podcasts on the market about and around SQL Server:\nSQL Down Under Podcast - Dr Greg Low Knee-Deep in Tech - Alexander Arvidsson SQL Server Radio - Guy Glanster \u0026amp; Matan Yungman Voice of the DBA\u0026rsquo;s podcast - Steve Jones DBAle - Two Chrises from Redgate BIFocal.show - John White \u0026amp; Jason Himmelstein Microsoft Research Podcast Update 23/01/2019 20:50\nI spotted it this morning after publication of this post. I must admit I\u0026rsquo;m really pleased that \u0026ldquo;Ask SQL Family\u0026rdquo; podcast has been awarded and it is in \u0026ldquo;Top 10 SQL Podcasts You Must Subscribe and Listen to in 2019\u0026rdquo;. Thank you for appreciating those efforts!\nEnjoy! Thanks for listening and reading!\nKamil\n","date":"2019-01-23T06:00:41Z","image":"/uploads/2019/01/prince-abid-653908-unsplash-1200.jpg","permalink":"/2019/01/seventeen-months-of-podcasting-recap/","title":"17 months of podcasting - recap"},{"content":"Hello folks! Good morning on Monday morning and let\u0026rsquo;s go through some interesting news from the #data world.\nPress Using Docker to run Integration Tests for dbachecks\nThis time, my friend Rob (T) presents how to run Unit Tests and Integration tests against multiple versions of SQL Server. All things using VS Code and Docker only.\n5 Reasons to Become an Apache Spark Expert\nAre you an Apache Spark guru? Here are 5 reasons why you should become one.\nMonitor your Azure Data Factory pipelines proactively with alerts (video)\nIn this video, Gaurav Malhotra presents a way of using integration events to create alerts.\nCathrine\u0026rsquo;s (T) 10 minutes video in a simple few steps explains how to use this new feature. BTW I love the fonts she used in the slides. Which one is that?\nHow to shoot yourself in the foot with Azure SQL Data Warehouse\nLook what kind of activities/decisions you should avoid making your loads faster.\nSQLBits conference needs more helpers\nAnnette (T) said: We are looking for more helpers for the Saturday at SQLBits. If you are interested - check the above link.\nAzure Data Factory Data Flow - Cheat Sheet - ver.1.01\nI updated my cheat sheet of ADF Data Flow on GitHub. Now it contains recently added transformations like Pivot, Unpivot and Window.\nA word from Microsoft\u0026rsquo;s CEO The necessity of tech intensity in today’s digital world https://t.co/hJFmfcF1I2\n— Satya Nadella (@satyanadella) 18 January 2019 Video: AI-powered robot for future Mars exploration https://www.youtube.com/watch?v=lEfE4MmmJAM\nSmile corner ","date":"2019-01-20T23:36:31Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/01/last-week-reading-2019-01-20/","title":"Last Week Reading (2019-01-20)"},{"content":"Good morning chaps! Bunch of news from the world of data freshly served.\nPress Office Hours 2019/1/2 (With Transcriptions)\nThe last episode. Personally, I\u0026rsquo;m sad, but waiting for new ideas from Brent (T).\nEasily embed secure Power BI reports in your internal portals or websites\nEmbed securely in internal applications with just a few button clicks.\nHow to Make DAX Distinct Counts 100x Faster with Exact Match Aggs\nRead if you love optimising your DAX code.\nOnline Book: Power BI From Rookie to Rockstar\nReza Red (T) shares over 1700 pages of his free books. Everything you want to know about Power BI.\nAzure DWH part 29: MicroStrategy Desktop with Azure\nRecipe of connection with Azure SQL Data Warehouse from MicroStrategy.\nGenerating SQL using Biml (T-SQL Tuesday #110)\nCathrine (T) proves that biml can generate not only SSIS packages.\nCreate Microsoft Flow Part 2- Face Recognition by Power Apss, Microsoft Flow and Cognitive Service\nLeila Etaati (T) is explaining full step-by-step process of building Face Recognition application.\nThe 2019 State of Database DevOps Report\nRedgate\u0026rsquo;s annual report has been published! Contains figures built based on over 1000 participants altogether.\nA word from Microsoft\u0026rsquo;s CEO Introducing new capabilities in Microsoft Teams to empower the more than 2 billion firstline workers around the world to do their best work https://t.co/tbISHOHp3y\n— Satya Nadella (@satyanadella) 10 January 2019\rVideo  Smile corner ","date":"2019-01-13T20:30:15Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/01/last-week-reading-2019-01-13/","title":"Last Week Reading (2019-01-13)"},{"content":"What popular tool has he created in 2006? How many people do live in our guest\u0026rsquo;s country? What can give you a valuable perspective?\nNew podcast episode is coming (next Friday) and in the meantime, you can watch that episode\u0026rsquo;s trailer.\nSimply click PLAY button above or move to the YouTube: https://youtu.be/ZmoA3ZhswAk\nEnjoy!\n","date":"2019-01-11T00:45:44Z","image":"/uploads/2019/01/ASF-018-video-invitation.jpg","permalink":"/2019/01/asf-018-video-invitation/","title":"ASF 018 video invitation"},{"content":"Azure Key Vault is a service which allows you to keep and manage passwords, certificates and other sensitive information securely. Out of the box we have capabilities to manage permissions having full control of who\u0026rsquo;ve got access to a particular type of information.\nWe can keep and securely manage 3 types of object:\nKeys Secrets Certificates Each item might be defined with a period of validity and activated with a future date if needed.\nIn this post, I gonna focus on Secrets, where you can store:\npasswords connection strings logins usernames tokens \u0026hellip; and others sensitive string To find out more - check Microsoft\u0026rsquo;s documentation page. (link)\nCreate new KeyVault Go to Azure Portal, find \u0026ldquo;Key Vaults\u0026rdquo;, click on it and then click \u0026ldquo;Add\u0026rdquo; button. You will see \u0026ldquo;Create key vault\u0026rdquo; section. Define your own name for creating vault, select (or create new) Resource Group and select Location.\nOnce new Key Vault is deployed - just refresh the list and click on the name - you will see fundamental information about the service - including: name, Location, Subscription name \u0026amp; ID, DNS Name and Pricing tier (SKU).\nCreate new Key We are interested in creating new \u0026ldquo;secret\u0026rdquo;, so go to \u0026ldquo;Secrets\u0026rdquo; and click \u0026ldquo;Generate/Import\u0026rdquo; button. We use \u0026ldquo;manual\u0026rdquo; method to input password, so choose this option, input a name (must be unique among this group in Azure Key Vault) and then: sensitive value. Optionally you can inform your team what is the content of that secret.\nBy default, a new key is active, enabled and has no expiration date but you can change it at any time.\nClick \u0026ldquo;Create\u0026rdquo; and your first password has been stored in your Azure Key Vault:\nAs you can see - all keys are versionable. All previous versions of items are kept, the latest version is the current one, but you always have access to all versions if require.\nGrant a permission You can control the access to Azure Key Vaults separately to each group, granting permissions to a various set of operation on them. Below table presents what rights you may granted.\nKey permissions Secret permissions Certificate permissions Get Get Get List List List Update Set Update Create Create Import Import Delete Delete Delete Recover Recover Recover Backup Backup Backup Restore Restore Restore Decrypt Manage contacts Encrypt Manage Certificate Authorities Unwrap key Get Certificate Authorities Wrap key List Certificate Authorities Verify Set Certificate Authorities Sign Delete Certificate Authorities Purge Purge Purge In order to grant permission(s), select \u0026lsquo;Access policies\u0026rsquo; from menu and you could see \u0026ldquo;Add new\u0026rdquo; button and list of objects being had permissions.\nFor instance, if you want to add the rights \u0026ldquo;Get\u0026rdquo; and \u0026ldquo;List\u0026rdquo; only for particular Azure Data Factory - simply check out two checkboxes in \u0026ldquo;Secret permissions\u0026rdquo; field and find an appropriate principal on the list (Select a principal). As you can see, the possibilities for granting rights go far beyond the standard \u0026ldquo;user\u0026rdquo; and \u0026ldquo;group\u0026rdquo;. You can grant permissions directly for applications or services created in Azure.\nicon-warning Bear in mind, that (as usual) pairs \u0026ldquo;permission-object\u0026rdquo; are based on GUID rather than names. Thus, once you delete the object and create a new one even though the same name - permissions must be reassigned.\nReading the secrets There are unlimited methods which allow you to read data from Key Vaults. Many (if not all) services and applications in Azure may get a password, connection string or token, including Azure Data Factory and Databricks. You can use any language (.NET, PowerShell, Python) you want to receive it.\nIn the following brief example - you can see how to read the secrets using PowerShell.\nImport-Module AzureRM.KeyVault\rConnect-AzureRmAccount\rGet-AzureRMContext\r$secret = Get-AzureKeyVaultSecret -VaultName \u0026#39;kv-sqlplayer\u0026#39; -Name \u0026#34;abc-database-password\u0026#34;\rWrite-Host \u0026#34;Secret Value is:\u0026#34; $secret.SecretValueText In the next posts, we will be using Key Vault very often.\nThanks \u0026amp; stay tuned.\n","date":"2019-01-09T08:00:53Z","image":"/uploads/2019/01/lock-2297578_1280_.jpg","permalink":"/2019/01/store-sensitive-information-with-azure-key-vault/","title":"Store sensitive information with Azure Key Vault"},{"content":"Good morning chaps! Bunch of news from data\u0026rsquo;s world freshly served.\nPress Introduction to Testing Your PowerShell Code with Pester\nRobert Cain introduces us Pester for making tests.\n9 popular ways to perform Data Visualization in Python\nRead this post if you interested in building charts with Python.\nNew automated machine learning capabilities in Azure Machine Learning service\nNow you can scale up your training jobs, use Azure Databricks or your local PC and use rich visual charts, forecasting or Model Explain Ability.\nAzure Data Factory Data Flow: Transform Data with Regular Expressions\nIn this post, Mark Kromer (T) shows how regular expression can be used in ADF to clean an address.\n2018: Community Influencer of the Year\nAaron Bertrand (T) announced his annual award and the winner is Bert Wagner (T)!\nCheat Sheets\nI have introduced a new page on this blog.\nA word from Microsoft\u0026rsquo;s CEO Nothing this time, sorry :)\nVideo: Robots inspired by Nature https://www.youtube.com/watch?v=drQP_0yaMXE\nSmile corner ","date":"2019-01-06T22:24:57Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2019/01/last-week-reading-2019-01-06/","title":"Last Week Reading (2019-01-06)"},{"content":"It is a third post on series related to installation and configuration of SQL Server installations.\nThis series is for professionals who start their journey with SQL Server administration and also for those who want to extend and structure their knowledge on SQL Server administration.\nPreparations We are starting with the following assumptions:\nAll steps described in this first blog post has been configured ISO file with SQL Server binaries and Cumulative Update files are available - steps how to download it are described in previous blog post There is a connection between Windows server and domain controller No issues with connectivity with Active Directory domain (no AD-related errors in System Event Log) Use case Imagine a situation when you are tasked to install a dozen SQL Server instances. Using an approach described in previous blog post installation can take a significant amount of resources and time, also there is a probability of errors due to typos (e.g. in directory paths, etc.)\nThere are two options that can be used for the installation:\nUsing ConfigurationFile.ini Using command line parameters to setup.exe file Below there are step-by-step instructions on how to install SQL Server using both options.\nInstallation of SQL Server using ConfigurationFile.ini In the previous post on one of the last configuration screen (Ready to Install), I have marked the path to ConfigurationFile.ini generated automatically by the installer. This file contains all of the settings that you have selected during installation using GUI.\nLet\u0026rsquo;s go to C:Program FilesMicrosoft SQL Server140Setup BootstrapLog\u0026lt;date_time\u0026gt;** location\nLet\u0026rsquo;s have a look at the ConfigurationFile.ini file:\nInside file we have several sections that are interesting for us:\nQUIET - determines if a user interface is disabled QUIETSIMPLE - Setup will display progress only, without any user interaction UIMODE - defines how User Interface will be displayed, requires that QUIETSIMPLE is set to True UpdateEnabled - allows installation of updates UpdateSource - specifies the location of updates INDICATEPROGRESS - specifies that the detailed log will be shown on the console Having all these parameters we can start installing SQL Server using different modes\nManual installation with populated configuration settings In this sections following parameters will be used:\nQUIET=\u0026ldquo;False\u0026rdquo; QUIETSIMPLE=\u0026ldquo;False\u0026rdquo; UIMODE=\u0026ldquo;Normal\u0026rdquo; UpdateEnabled=\u0026ldquo;False\u0026rdquo; UpdateSource=\u0026ldquo;MU\u0026rdquo; INDICATEPROGRESS=\u0026ldquo;False\u0026rdquo; To start installation run command line and provide the following command:\nE:setup.exe /ConfigurationFile=D:DBAConfigurationFile.ini where E: is the path to SQL Server binaries.\nAfter running above command, GUI installation with start with values already provided\nAll of the configuration values are taken from ConfigruationFile.ini.\nManual installation with populated configuration settings using latest Cumulative Update located in D:DBAUpdates directory In this sections following parameters will be used:\nQUIET=\u0026ldquo;False\u0026rdquo; QUIETSIMPLE=\u0026ldquo;False\u0026rdquo; UIMODE=\u0026ldquo;Normal\u0026rdquo; UpdateEnabled=\u0026ldquo;True\u0026rdquo; UpdateSource=\u0026ldquo;D:DBAUpdates\u0026rdquo; INDICATEPROGRESS=\u0026ldquo;False\u0026rdquo; To start installation run command line and provide the following command:\nE:setup.exe /ConfigurationFile=D:DBAConfigurationFile.ini where E: is the path to SQL Server binaries\nAfter running above command, GUI installation with start with values already provided.\nAll of the configuration values are taken from ConfigruationFile.ini.\nAutomated installation using latest Cumulative Update from D:DBAUpdates directory with progress indication In this sections following parameters will be used:\nQUIET=\u0026ldquo;False\u0026rdquo; QUIETSIMPLE=\u0026ldquo;True\u0026rdquo; ;UIMODE=\u0026ldquo;Normal\u0026rdquo;\u0026lt;- commented out UpdateEnabled=\u0026ldquo;True\u0026rdquo; UpdateSource=\u0026ldquo;D:DBAUpdates\u0026rdquo; INDICATEPROGRESS=\u0026ldquo;False\u0026rdquo; To start installation run command line and provide the following command:\nE:setup.exe /ConfigurationFile=D:DBAConfigurationFile.ini /IAcceptSQLServerLicenseTerms where E: is the path to SQL Server binaries\nAfter running above command, the installation will start using values from the configuration file.\nAfter setting up all installation file, installation of SQL Server itself will start.\nAfter completing this step SQL Server is ready for use.\nFully automated installation using latest Cumulative Update from D:DBAUpdates directory In this sections following parameters will be used:\nQUIET=\u0026ldquo;True\u0026rdquo; QUIETSIMPLE=\u0026ldquo;False\u0026rdquo; ;UIMODE=\u0026ldquo;Normal\u0026rdquo;\u0026lt;- commented out UpdateEnabled=\u0026ldquo;True\u0026rdquo; UpdateSource=\u0026ldquo;D:DBAUpdates\u0026rdquo; INDICATEPROGRESS=\u0026ldquo;True\u0026rdquo; To start installation run command line and provide the following command:\nE:setup.exe /ConfigurationFile=D:DBAConfigurationFile.ini /IAcceptSQLServerLicenseTerms Where E: is the path to SQL Server binaries\nThere will be no GUI window displayed, the only information is shown on the console. Once all steps complete SQL Server is ready for use.\nInstallation of SQL Server using a command line An alternative option is to use command line parameters to install SQL Server. In the following example, I am going to install SQL Server with indication of installation.\nFor the installation following parameters are provided:\nQS ACTION=Install FEATURES=SQLENGINE ENU UpdateEnabled=1 UpdateSource=\u0026ldquo;D:DBAUpdates\u0026rdquo; INSTANCEDIR=\u0026ldquo;D:MSSQLServer\u0026rdquo; INSTANCENAME=MSSQLSERVER SQLSVCACCOUNT=\u0026ldquo;DATACOMMUNITYSQLServerEngine$\u0026rdquo; SQLSYSADMINACCOUNTS=\u0026ldquo;DATACOMMUNITYadministrator\u0026rdquo; AGTSVCACCOUNT=\u0026ldquo;DATACOMMUNITYSQLServerAgent$\u0026rdquo; AGTSVCSTARTUPTYPE=\u0026ldquo;Automatic\u0026rdquo; BROWSERSVCSTARTUPTYPE=\u0026ldquo;Disabled\u0026rdquo; INSTALLSQLDATADIR=\u0026ldquo;D:MSSQLServer\u0026rdquo; SQLBACKUPDIR=\u0026ldquo;D:SQLBackup\u0026rdquo; SQLCOLLATION=\u0026ldquo;SQL_Latin1_General_CP1_CI_AS\u0026rdquo; SQLSVCSTARTUPTYPE=\u0026ldquo;Automatic\u0026rdquo; SQLTEMPDBDIR=\u0026ldquo;D:SQLTempDB01\u0026rdquo; \u0026ldquo;D:SQLTempDB02\u0026rdquo; \u0026ldquo;D:SQLTempDB03\u0026rdquo; \u0026ldquo;D:SQLTempDB04\u0026rdquo; SQLTEMPDBLOGDIR=\u0026ldquo;D:SQLTLog\u0026rdquo; SQLUSERDBDIR=\u0026ldquo;D:SQLData00\u0026rdquo; SQLUSERDBLOGDIR=\u0026ldquo;D:SQLTLog\u0026rdquo; SQLTEMPDBFILECOUNT=4 IACCEPTSQLSERVERLICENSETERMS Having all parameters together, command will look like as follows:\nE:setup.exe /QS /ACTION=Install /FEATURES=SQLENGINE /ENU /UpdateEnabled=1 /UpdateSource=\u0026#34;D:DBAUpdates\u0026#34; /INSTANCEDIR=\u0026#34;D:MSSQLServer\u0026#34; /INSTANCENAME=MSSQLSERVER /SQLSVCACCOUNT=\u0026#34;DATACOMMUNITYSQLServerEngine$\u0026#34; /SQLSYSADMINACCOUNTS=\u0026#34;DATACOMMUNITYadministrator\u0026#34; /AGTSVCACCOUNT=\u0026#34;DATACOMMUNITYSQLServerAgent$\u0026#34; /AGTSVCSTARTUPTYPE=\u0026#34;Automatic\u0026#34; /BROWSERSVCSTARTUPTYPE=\u0026#34;Disabled\u0026#34; /INSTALLSQLDATADIR=\u0026#34;D:MSSQLServer\u0026#34; /SQLBACKUPDIR=\u0026#34;D:SQLBackup\u0026#34; /SQLCOLLATION=\u0026#34;SQL_Latin1_General_CP1_CI_AS\u0026#34; /SQLSVCSTARTUPTYPE=\u0026#34;Automatic\u0026#34; /SQLTEMPDBDIR=\u0026#34;D:SQLTempDB01\u0026#34; \u0026#34;D:SQLTempDB02\u0026#34; \u0026#34;D:SQLTempDB03\u0026#34; \u0026#34;D:SQLTempDB04\u0026#34; /SQLTEMPDBLOGDIR=\u0026#34;D:SQLTLog\u0026#34; /SQLUSERDBDIR=\u0026#34;D:SQLData00\u0026#34; /SQLUSERDBLOGDIR=\u0026#34;D:SQLTLog\u0026#34; /SQLTEMPDBFILECOUNT=4 /IACCEPTSQLSERVERLICENSETERMS After running above command, installation will start\nAfter installation, the progress window will be closed and SQL Server is ready for install.\nThis option my preferred way to install SQL Server. If you are interested in PowerShell script for unattended SQL Server installation based on provided parameters, visit my GitHub repository.\nSummary In complex environments installation of SQL Server instances can be a tedious task that can be easily automated. This blog post extensively describes methods of possible installation.\nAlso, smaller environments can benefit from this approach by minimizing the time needed for installation, keeping quality on the same level. Every time when you want to use unattended installation SQL Server, check this blog post for all steps.\nLinks More information about steps described above can be found on the following sites:\nInstall SQL Server using a configuration file\nInstall SQL Server from the Command Prompt https://github.com/michalsadowski/SQLBuild\n","date":"2019-01-03T08:00:58Z","image":"/uploads/2019/01/autos-technology-vw-multi-storey-car-park-63294.jpeg","permalink":"/2019/01/unattended-installation-of-sql-server/","title":"Unattended installation of SQL Server"},{"content":"Good morning #sqlfamily folks!\nThe very first post in the new year 2019. Tell me: when this time passed??? The year 2018 has just gone. From the blog perspective, it was the first, full year of posting.\nIt\u0026rsquo;s about time to wrap up the statistics of the year. I mean\u0026hellip; not in numbers but in popularity of blog posts. Let\u0026rsquo;s see what articles were visited the most. Surprisingly (at least for me), series of the podcast is not in the top 10*.\nSCD type 1 \u0026amp; type 2 in MERGE statement Slowly Changing Dimensions (SCD) New DB project with SSDT and import existing database #TSQL2sDay – How SQLPackage.exe can spoil your deployment Azure Data Factory v2 and its available components in Data Flows sp_send_dbmail fails with query result attached as file How to create DACPAC file? Setting up Code Repository for Azure Data Factory v2 Functions in the USQL – the hidden gem in the Summer 2017 Update Preparation for SQL Server installation * Ranking based on Unique Page Views from Google Analytics.\n","date":"2019-01-01T11:00:48Z","image":"/uploads/2018/12/open-fire-3879031_1280.jpg","permalink":"/2019/01/the-most-read-posts-in-2018/","title":"The most-read posts in 2018"},{"content":"Good day SQL folks!\nHere is your weekly list of articles worth to read! The year is almost ended so I would like to wish you all the best in the new year! We have SQL Server 2019 with all exciting features, we have the Azure Cloud with new \u0026ldquo;stuff\u0026rdquo; that will be announced soon!!!\nBut now, straight to the point - today I give you a lot of Azure Cosmos DB articles to read!\nOptimizing for development and testing in Azure Cosmos DB\nHow to start Azure Cosmos DB development for free.. You have three options to do it without any investment on your side.\nAzure Cosmos DB and multi-tenant systems\nBuilding a multi-tenant system on another multi-tenant system can be challenging, but Azure provides us all the tools to make our task easy. An example of a multi-tenant system would be a company providing background check services that any other company can use in their HR system.\nWorking with Azure Cosmos DB stored procedures, triggers, and user-defined functions\nAzure Cosmos DB provides language-integrated, transactional execution of JavaScript. When using the SQL API in Azure Cosmos DB, you can write stored procedures, triggers, and user-defined functions (UDFs) in the JavaScript language. You can write your logic in JavaScript that executed inside the database engine.\nOptimize the cost required to read and write data from Azure Cosmos DB\nThis article describes how the cost required to read and write data from Azure Cosmos DB is calculated. Read operations include get operations on items and write operations include insert, replace, delete, and upsert of items.\nTotal Cost of Ownership(TCO) with Azure Cosmos DB\nAzure Cosmos DB is designed with the fine grained multi-tenancy and resource governance. This design allows Azure Cosmos DB to operate at significantly lower cost and help users save. Currently Azure Cosmos DB supports more than 280 customer workloads on a single machine with the density continuously increasing, and thousands of customer workloads within a cluster.\nCheers,\nDamian\n","date":"2018-12-31T06:32:04Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/12/last-week-reading-2018-12-30/","title":"Last week reading (2018-12-30)"},{"content":"The 2018 year is coming to the end. It\u0026rsquo;s a good time to review the passing year and\u0026hellip; check whether you put on the blog all events pictures you have promised. And start checking it since 2017... I found out that I have some gaps in terms of pictures and these gaps are mainly per 2017 as 2018 was being updated quite regularly.\nSo, here you have new photo albums added with a date back from 2017 (and one from this year in Manchester).\nMaybe you will find yourself among these pictures?\n14/07/2018 SQL Saturday #730 Manchester 14/10/2017 SQL Saturday #642 Sofia 07/10/2017 SQL Saturday #656 Denmark 20/05/2017 SQL Saturday #616 Kiev 25/02/2017 SQL Saturday #589 Pordenone Find out more in PHOTO category.\n","date":"2018-12-26T23:41:04Z","image":"/uploads/2018/12/sqlsaturdays-2017.png","permalink":"/2018/12/some-pictures-from-2017s-events/","title":"Some pictures from 2017's events"},{"content":"The Christmas is coming. But before we start the festive time and sit down to the Christmas Eve dinner - a few articles to feed up our brains (before we feed up our tummies).\nMerry Christmas to everyone!\nKamil\nPress Comparison Review: Microsoft SSDT vs Redgate SQL Source Control\nInterested in DevOps for MSSQL databases? Check this comparison prepared by Alex (T) and published by Brent (T).\nWhy DataOps and not just DevOps?\nCode and Data are very different - Terry McCann (T) is explaining.\nUsing Biml to automagically keep your external polybase tables in sync!\nBen (T) presents a way of reducing the pain of setting up and maintain external tables.\nautomated data masking\nChrissy LeMaire (T) and Sander Stad (T) present a new data protection feature in dbatools.\nPower BI And Column Order In M Queries And Dataset Tables\nChris (T) shows a scenario when column order does matter.\nMicrosoft Power BI roadmap\nWhich direction the product goes in the year 2019.\nAzure Data Factory Data Flow\nJames Serra\u0026rsquo;s (T) observations about #ADFDF.\nCumulative Update #13 for SQL Server 2017 RTM\nTime to update?\nA word from Microsoft\u0026rsquo;s CEO 10 change-makers who inspired me in 2018 https://t.co/kQt8nBVCjF\n— Satya Nadella (@satyanadella) 18 December 2018 Video PITTA - Transformative Autonomous 4K Selfie Drone\nhttps://www.youtube.com/watch?v=-ogRJZvhZVM\nSmile corner ","date":"2018-12-23T17:32:12Z","image":"/uploads/2018/12/santa-claus-reading-letters-ftr.jpg","permalink":"/2018/12/last-week-reading-2018-12-23/","title":"Last Week Reading (2018-12-23)"},{"content":"It is a second post on series related to installation and configuration of SQL Server installations.\nThis series is for professionals who starts their journey with SQL Server administration and also for those who want to extend and structure their knowledge on SQL Server administration.\nPreparations We are starting with following assumptions:\nAll steps described in previous blog post has been configured There is connection between Windows server and domain controller No issues with connectivity with Active Directory domain (no AD related errors in System Event Log) There is Internet access for downloading ISO file with binaries of latest SQL Server - in our case we are using SQL Server 2017 Developer Edition, latest Cumulative Update As a first step we are going to download SQL Server 2017 Developer Edition\nHow to download SQL Server 2017 Developer Edition? Open your favorite web browser and go to https://www.microsoft.com/en-us/sql-server/sql-server-downloads website\nClick on Download now and run file downloaded\nSelect Download Media\nSelect ISO file and provide path where file will be download and click Download\nWait till file is downloaded and copy ISO file to server where SQL Server will be installed.\nISO file with SQL Server 2017 has been successfully downloaded.\nHow to install SQL Server 2017 Developer Edition? Mount file or double click to open ISO file.\nBrowse to DVD drive and run setup.exe\nClick on Installation section\nClick on New SQL Server stand-alone installation or add features to an existing installation\nSelect free edition or provide Product Key\nNote: All of the editions (Developers/Enterprise/Standard etc.) of SQL Server are using the same ISO file - the only difference is in the Product Key that is stored in ISO:\\x64\\DefaultSetup.ini\nRead and accept the license terms\nReview if all rules passed\nIf server has access to Internet (it shouldn\u0026rsquo;t) you can select option to check updates on Microsoft website and click Next.\nYou will receive error that setup couldn\u0026rsquo;t contact with Microsoft Update service - don\u0026rsquo;t worry and click Next.\nOn the Install Rules screen check if all rules are in Passed state, review any Warnings and click Next\nOn Feature Selection screen select Database Engine Service and update Instance root directory\nOn Instance Configuration screen select Default instance and click Next\nOn Server Configuration screen update accounts for SQL Server Agent and SQL Server Database Engine. Make sure that Grant Perform Volume Maintenance Task privilege to SQL Server Database Engine Service\nMake sure that you have provided managed service account created in previous blog post\nClick on Collation tab and check if correct collation is selected. If there is incorrect collation you can click Customize\u0026hellip;\nYou can use Windows collation or SQL collation and click OK\nOn Database Engine Configuration select Windows authentication mode and Add Current User. Click on Data Directories\nProvide correct Data root directory, User database directory, User database log directory and Backup directory\nYou can click on [\u0026hellip;] and browse for directory.\nClick on TempDB tab and update number of files (my server is configured with 4 visible threads). Update Initial size and Autogrowth for data files to 4 GB. Add Data directories and update Log directory. Update Initial size and Autogrowth for log file to 2 GB and click Next.\nOn information screen about increased setup time due to large log file size click Yes.\nOn Ready to Install make note of the path to Configuration file and click Install\nWait till installation completes\nOn Complete screen review Summary log file and click on Close\nAbove steps describes installation process of SQL Server.\nHow to install latest Cumulative Update for SQL Server 2017? Browse to https://support.microsoft.com/en-us/help/4047329/sql-server-2017-build-versions and download latest cumulative update.\nOnce file is downloaded run installer of Cumulative Update\nOn License Terms accept the license terms and click Next\nOn Select Features select all instances and click Next\nOn Check Files In Use screen click Next\nOn Ready to update screen click Update\nWait till installation completes\nOn Complete screen review Summary log file and click on Close\nAfter all above steps SQL Server is installed with latest Cumulative Update.\nSummary Installation of SQL Server is one of the most important step before promoting server to normal operation. Installation of latest Cumulative Update helps solving many performance and manageability issues. Every time when you install SQL Server, check this blog post for all steps.\nLinks More information about described above steps can be found on following sites:\n/2018/12/preparation-for-sql-server-installation/\nhttps://www.microsoft.com/en-us/sql-server/sql-server-downloads\nhttps://support.microsoft.com/en-us/help/4047329/sql-server-2017-build-versions\nhttps://technet.microsoft.com/en-us/library/ff803383.aspx\nhttps://support.microsoft.com/en-my/help/2154845/recommendations-to-reduce-allocation-contention-in-sql-server-tempdb-d\n","date":"2018-12-20T07:00:07Z","image":"/uploads/2018/12/installation.jpg","permalink":"/2018/12/installation-of-sql-server-using-gui/","title":"Installation of SQL Server using GUI"},{"content":"Hello data geeks. Last week we had many news about Cosmos DB, and this time the top topics are Power BI, Python, Azure Functions and ADF. We have new sections starting from today including Video and Smile Corner. Have a good Monday and entire week!\nKamil\nPress Announcing new AI Capabilities for Power BI to make AI Accessible for Everyone\nLearn how to build and integrate your Azure Machine Learning models with Power BI.\nMultiple Layers of Aggregations in Power BI; Model Responds Even Faster\nReza Red explains multiple levels of aggregation in Power BI.\nTaking a closer look at Python support for Azure Functions\nPython in Azure Functions\u0026hellip;\nAzure Functions now supported as a step in Azure Data Factory pipelines\n\u0026hellip; and Azure Functions in ADF.\nPower BI Desktop December 2018 Feature Summary\nList of features introduced this month to Power BI Desktop.\nPower BI Dataflows and Azure Data Lake Storage Gen2 Integration Preview\nMore about Common Data Model (CDM) and Azure Data Lake Storage.\nDynamic Legend in Power BI Visual, Line Chart\nPrathy presents quick post about dynamic slicer with multiple selections.\nUsing Azure Data Factory v2 Activities \u0026amp; Dynamic Content to Direct Your Files\nGet Metadata, ForEach, and Copy activities in action.\nA word from Microsoft\u0026rsquo;s CEO A powerful example of how teaching computer science can change how students approach learning – and, in the process, provide them with the skills they need to thrive in any career. https://t.co/30FoLIS3ZY\n— Satya Nadella (@satyanadella) 4 December 2018 Video Microsoft HoloLens: Partner Spotlight with Ford\nhttps://www.youtube.com/watch?v=3QyA7HhIYkg\nSmile corner ","date":"2018-12-16T21:40:05Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/12/last-week-reading-2018-12-16/","title":"Last week reading (2018-12-16)"},{"content":" Introduction Alex Whittles is the owner and principle consultant at Purple Frog, a SQL Server Business Intelligence consultancy in the UK with multinational clients in a variety of sectors. He specialises in all aspects of dimensional data modelling, data warehousing, ETL and cubes using the SQL Server stack. Alex has an MSc (Master of Science) in Business Intelligence, is a chartered engineer, and a member of Mensa. Community leadership includes being on the SQLBits committee, Director of SQL Relay, founder and leader of the Birmingham Data Platform user group. Alex is also a regular speaker at many SQL Server events around the world including SQL Relay, SQLBits, SQL Saturdays, 24 HOP and the PASS Summit.\nThis talk has taken place during Data Relay (formerly SQL Relay) conference in Reading (UK), on 11th October 2018 (Thursday). Interviewers: Kamil Nowinski.\nWhich unusual mean of transportation Alex use from time to time? Why monitoring of data quality does matter and what is the very efficient contrary of SCD in SSIS in BI loading process? Which skill is important if you want to jump into the IT market? Find out the answers on these questions and much, much more.\nAudio version [row][third_paragraph][/third_paragraph][paragraph_right] Don\u0026rsquo;t you have time to read? You can listen this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3.\nEnjoy! [/paragraph_right][/row]\nTranscript Kamil Nowinski: Hi Alex. Thanks for being a guest of this podcast and accepting my invitation. Could you, at the beginning, tell me what your name is and where you live?\nAlex Whittles: Good afternoon! Thanks for inviting me. My name’s Alex Whittles and I live in Telford in Shropshire.\nKN: I know that you have your own consultancy company. What are you doing for a living exactly?\nAW: So I run Purple Frog, which started 12 years ago now and it’s a Microsoft BI consultancy. Initially, we started focusing on data warehousing, ETL, data modeling and cubes with a special focus on cubes. But over the years, as the Microsoft data platform has evolved and expanded, that’s grown in breadth into Power BI, Databricks, Data Lake, machine learning, quite a lot different topics now. But still focused around the concept of taking a vast amount of data from around organizations and putting it into an easy, simple way of accessing useful information and spreading it around the business. And the tools evolve and change over the years, but the need of a company to easily access its own data in a reliable, consistent way – that still the same as it always has been and that’s what we do as a business.\nKN: Do you build any framework for example to help yourself achieve the goals faster?\nAW: So we have a number of different methods of what we do, different approaches. One is pure consultancy, where we go into a business and just talk to them about opportunities, what they can do, how they can use technology to solve a problem. Advise them, mentor their staff, provide training advice and hand-holding to help them achieve their goals and put them in the right direction. Then we have a development arm where we build complete end-to-end solutions and systems internally and yeah, we’ve spent a lot of the last 12 years building up quite a comprehensive framework of code that helps us build much more highly scalable robust solutions. So if you look at a standard data warehouse and ETL solution, probably about 80% of the code is reused project to project to project. We’re talking about restart ability, robustness, logging, error handling, sequencing control, management – all this kind of stuff.\nKN: And basically, it doesn’t matter if you use the old-fashioned ETL or the new, unlike in Azure Data Factory stuff?\nAW: Exactly. The concept’s the same. The actual metadata, the information you need and the approach to it is still very, very similar. Okay, we move from ETL to ELT or ETLT, various different combinations, but the concept is there, and the underlying functionality is still very, very similar. So we’ve spent a lot of time and resources building up a large framework and code automation set. That’s when we define a particular user or particular customer’s requirements and work out what they need, the complex job is doing the requirements gathering, the source to target mapping, where the data coming from. Once we identify that metadata, then the process of actually building the code should be very straightforward and so our framework automates a lot of that code build, which means it’s very standardised reusable code. Very, very consistent, so any of our developers can pick up any code than any other developer’s done and know exactly how it works. But also, our customers get a very consistent approach to the code that’s being developed. And that allows us to build much bigger solutions. We’re actually quite a small team. We’re a team of six people, but we’ve done multinational huge corporations products for them that you would think would take teams of fifty or a hundred. And because of a lot of this code automation of the underlying framework, we can do very, very large projects with actually quite a small team.\nKN: Exactly, that was my next question, that you and your company basically are the same. I’ve seen that you’re hiring people and so you mentioned that the size of your company is six people, yeah?\nAW: Well, is actually five today, it’ll be six on Monday.\nKN: Okay, great, congratulations, new member of the team.\nAW: We’re growing as fast as we can. The main limitation we have is availability of staff and resources. So we’re always on the lookout for good people and that could be existing senior people that are already skilled in technology that can come in on a consultant level, or it could be junior staff that we can train up and we have a very strong ethos in the company of training, mentoring and growing skills with the pace of change of the technology in the Microsoft Data Platform world, it’s no good if you’re an expert in technology today, because it’s going to change tomorrow, so we have to keep evolving and learning new technology and probably 20% of our time in the office is spent exploring new technologies as a constant iterative learning process, and so even our experienced staff have to keep learning, have to keep playing with new technologies doing proof-of-concept projects. We’re pretty open to taking on anybody, no matter what your skill level, as long as you have the desire to learn, the ability to learn and as long as you enjoy working with this tech. If you don’t enjoy working with it, you’re not going to do a good project. We want people who really embrace and enjoy this kind of work and enjoy playing with data. And that takes a special kind of person.\nKN: It’s amazing that you are answering my questions in advance basically. You have answered three of my questions.\nAW: It’s all machine learning predictive models. We plan for it in advance.\nKN: So yes, you’re still looking for new people to your company, that’s good. Different level of experience of them, yeah?\nAW: Yeah, so the last few people we’ve taken on, the guy’s starting on Monday, it’s focusing around Power BI and Pyramid Analytics, but with an underlying core of data modelling, data warehousing and ETL, but focusing a lot on the actual presentation of that into customers. The previous chap who started, his main focus is on machine learning, so his background degree was in artificial intelligence machine learning models and he’s really pushing that side of the business for us. The guy before that is actually a C# developer, so even though we’re a business intelligence house, we are finding there’s a lot more coding coming into our world and so we’ve now got a C# person and we’re looking for another one actually as well.\nKN: So C# helps you in SSIS scripts and also in U-SQL, yeah?\nAW: Yeah, it’s not just C#. It’s C#, its Python, it’s R, there’s a lot of code creeping into our world these days.\nKN: More and more languages.\nAW: When you look at tools like Databricks, you’ve got Scala, you’ve got Python in there, you need to have a lot of languages.\nKN: Maybe in the nearest future you will not need and know these languages because there is a new feature in Azure Data Factory like Data Flows.\nAW: Absolutely. But then another part of Purple Frog is that we also provide a managed service. So if you’ve got a business intelligence solution with a Data Warehouse and an ETL system, a cube, then as a business you need that to be up and running and maximize uptime, because it becomes a business critical solution. If your board or CxOs are relying on this system to produce all of their management KPIs and their understanding of the performance of the business, when that’s down for two days, it can cause you problems. Some of our customers rely on it hour by hour during the day in a real-time environment to manage their real-time stock distribution and manufacturing warehouses. Banking customers need real-time information from their warehouse to actually manage their investments and portfolios. So a good data warehouse and a good cube or reporting solution will very quickly become a business critical system. And so we provide a monitoring system where we look after those solutions for customers and monitor them in real time on a second-by-second basis and monitor the data quality, the data accuracy as well as the actual functionality and successful completion of the data load jobs and everything else that goes with it. Even the duration of how long the load takes, how long the cube’s taking, query duration, that kind of thing. So we have quite a comprehensive monitoring solution that we built in-house using C# that just sits there, constantly checking all the time in your servers, and as soon as there’s a potential problem, it alerts us with our support desk and we can jump in straight away and actually help fix that for you straight away, before you even know there’s our problem. So things like that, it’s not actually a BI solution but it’s a business critical C# solution that we’ve developed, that help us manage and maximize the uptime of our customers’ environments. So everywhere we look there’s code, there’s C#, things like that coming into it.\nKN: From the business perspective, you can save a lot of money using those solutions.\nAW: Absolutely. So we had a choice of buying an off-the-shelf solution or numerous off-the-shelf solutions that are available. Most of the vendors of monitoring software tend to focus more on the DBA side of the world. And there’s some fantastic tools out there that monitor databases from a DBA perspective, but from a BI perspective, it’s not necessarily enough just to be able to look at how fast your queries are running in the data warehouse. How fast your query is running in the cube. It’s also really critical to look at is the data right. Has something changed in your data that isn’t right. It may be working technically, but is the actual result coming out correct. So things like in a retail environment. We know that your retail sales for last year shouldn’t change. Therefore, we monitor every five minutes. We run a check to say “OK, the volume of sales last year was this value”. As soon as that changes\u0026hellip; any range you define. We work with customers to define specific or customized checks and so, let’s say we’re looking at last year’s sales. We know that we’re already in October though, so last year’s sale shouldn’t change. Every five minutes we’ll check that number and make sure that’s consistent in the cube. That never changes. It means you’ve got some rogue data coming in to the system. Transactions for this week have gone against the wrong year. Something’s gone wrong in that chain of process. The data’s loaded okay, but it’s actually not valid data, and so we check the data quality. Things like year-on-year performance or year-on-year changes, we can run a minute-by-minute checks or hourly checks to say “okay, the value of sales in this week should be roughly equivalent to the comparable week last year, plus or minus 10% growth for example”. And as long as your data falls within that region, that boundary, we know that it’s pretty much gonna be okay. As soon as it falls outside the boundary, it indicates there may be some rogue data or something’s happened with invalidated environment. And so it’s both about technical functionality, performance and also data quality. Can our customers rely on their cube to give them accurate data. And so it’s a very specific business intelligence orientated monitoring service rather than purely looking at locks and blocks and weights in a database.\nKN: That’s an interesting approach to monitoring basically, how to monitor the quality of data.\nAW: It monitors not only the technical side of it, but the confidence level. We can find these issues and resolve them before most businesses even know they exist. And so by building up, as we get a managed service contract with the customer, we work with them over time to constantly build this set of checks to add more and more over time to improve the robustness and reliability of their solutions. That’s a constant growing involvement and yeah, it’s interesting.\nKN: OK, so our listeners and readers can’t see it but currently we are sitting at Microsoft HQ in Reading. This is the fourth day of Data Relay. Tell me something about this event.\nAW: So Data Relay, which until this week was called SQL Relay, started 9 years now as a brainchild of two fantastic people who have been long-term founding members of the Microsoft SQL Server Community Tony Rogerson and Chris Testa-O’Neill, who have been instrumental in shaping the whole SQL Server Microsoft Data Platform community in the UK. Both of them long-standing MVPs, Chris now a Microsoft employee. They were discussing, I think they were over at the MVP summit in Seattle ten years ago, and came with the concept of SQL Relay as a community event. So in the UK, we’re very, very lucky to have a large number of Microsoft data events. We have SQLBits, which is the second biggest conference of its kind in the world, and we have a lot of user groups or meetups. They run all around the country in the evenings over a couple of hours. And there wasn’t really anything in between, so SQLBits is once a year, huge events, a couple thousand people, great event. And it just keeps getting better and better, we love it. But there was nothing really in between. And a lot of the user groups, we’re operating entirely independently and not really communicating too well as a whole about coordinating speakers, approach to things and we’re just working together. So Chris and Tony came up with the concept of SQL Relay to really bring the Data Platform community leaders together to run a centralized event. And it started out in its first year as a coordinated set of evening user groups over a week, across the entire UK, culminating in an event in Thames Valley Park in Reading here, where we are today. And if I remember correctly, we had the amazing Itzik Ben-Gan doing a keynote speech on introducing Windowing Functions to SQL Server and our minds were just blown away. And it worked, it got all of the community leaders working together and discussing how we can actually evolve the Microsoft community in the UK and how we can improve it to make it bigger and better. And so the second year SQL Relay ran again and instead of just being evening news groups, we extended it to all-day events. They were spread over two weeks. And so we had more speakers involved, we managed to get some funding from Microsoft and it turned into a full-day single-track event and it just grew. And there was an opportunity to really evolve it and turn it into a really powerful feature in the UK data community calendar. So I was involved from year one, obviously I run the Birmingham user group, so I was involved from that perspective. I think the first year Tony took the lead as chair of SQL Relay to really coordinate the entire event. Second year, if I remember correctly, it was Chris who took over, Chris Testa-O’Neill is the lead. And then in year 3 they handed the baton to me. So I took over as the chair of SQL Relay and yeah, took it over and ran it for a year. And it was a really fascinating challenge for, ultimately, a data geek. I play with databases and I run my own business, so I have that side of it with the management and the financial management and everything else that goes with it. But running a conference is a whole different set of challenges. I took it on without really realizing the extent of how difficult it was going to be, and time-consuming, but incredibly rewarding and I loved it. Really enjoyed it. And so in that year 3 we expanded and we still had two weeks worth of events, so I think we covered eight venues across the UK over a two-week period. And each event we had a couple of different tracks, so two rooms in parallel with various speakers coming in throughout there. And it was a huge success.\nKN: When it was, the eight days?\nAW: Now you’re stretching my memory, crikey! I know we did a Scottish venue. I can’t remember, whether it was Glasgow or Edinburgh, we did both of those in years three and four, I can’t remember which way round it was. I’ve got a feeling it may have been Glasgow that year. We did, I’ve got a feeling, Newcastle, Leeds, Birmingham, Reading, London, Cardiff, Southampton\u0026hellip; I can’t remember where else. That’s a good starting point anyway. Something like that anyway, and so I got more and more involved in that year obviously with leading it and leading the sponsorship and getting more sponsors involved to help fund it. And over the years, the committee members have changed, a lot of people have come into the committee, a lot of people have left, a lot of people have joined Microsoft, I think three SQL Relay committee members have now joined Microsoft. And so the people involved have changed, but there’s always been other community leaders that run user groups. And one of our very initial reasons for doing it, was to help promote user groups in the community in the UK. And that’s all very much a part of it. Every keynote or introduction, we’re always trying to promote other user groups. I was actually talking to a gentleman earlier today who lives in Northampton. And there’s currently no meetup in Northampton so he’s quite keen to get involved in the community, so I’m now talking to him about how we go about setting up a new user group there. It’s about what we need to do. It’s setting up a venue, how to promote it, how to get the speakers, getting contacts, that kind of thing, venues don’t come for free, we like to put on catering if you can, get some pizzas in or something and that all takes a bit of money, so which sponsors are around that can actually help with that, what kind of venue do you need, how do you go about it. So when I set up the Birmingham venue, Tony Rogerson did the same for me. He gave me a lot of advice on how to go about setting up the event. And now he passed the baton on and I’ve had a few years’ experience, so it’s really nice to be able to help out other people. And that’s one of the nice things about the UK. Not just UK, the whole worldwide Microsoft community is everyone wants to help. Everyone wants to pass on information and knowledge and work together to make it happen. And that is really typified by SQL Relay or Data Relay now. It’s a bunch of people who care about the technology, who care about wanting to promote the technology and want to help other people learn about it. And if that can help user groups and meetups as well, encourage new speakers to come through, give them a platform to speak on. There’s a couple of speakers this week who have never spoken before on a big stage and it’s really nice to be able to give them a platform to help encourage and develop their own speaking skills, to get new speakers coming through talking about new topics or even existing topics, but they’ve got a different way of presenting it. They’ve got a different set of experience or they solve different problems.\nKN: And this this works especially for the SQL Server area, we call it SQL family. It works perfectly I think.\nAW: That’s what it’s about. It’s about always working together to help each other learn. I was very lucky early on in my career when I was trying to work out how to build these solutions early on when I was first starting. I never had an employer that would pay for training and so I relied very heavily on people who would give their time to write blog posts or to talk at conferences. SQLBits to me was a such an important conference to go and attend to learn how to do things and to learn from the likes of Chris Webb for example who’s speaking here today and I’ve just seen him do a session on M. Fantastic session. I learned a lot of what I know about cubes from watching Chris do talks and reading blog posts and talking to him about it. And a lot of the stuff we learn as a community is shared freely amongst each other in blogs, in books, in conferences. Also I learned a lot from going to these conferences, it’s really nice now to be in a position where I can help feed that back and help pass on that knowledge to the next generation and help now run the conferences, and also talk about technology as well.\nKN: That question would be very good for the next one. So what hints would you give to young people who wanted to start working on the IT market.\nAW: It’s very challenging. Back when I started, that was some time ago. The Microsoft Data Platform world was actually a relatively well-defined small world. It was SQL Server, it was Integration Services. Actually, it was DTS, but I won’t go back that far. It was Analysis Services or OLAP Services as it was, and that was really it. The challenge of any project wasn’t really determining the architecture, it was understanding the customers’ requirements and building the data model. That’s why a lot of my focus is all around data modeling, because so many problems in a project happened because of poor data modeling and that was always the core focus of my work. When you’re getting into this technology now, it’s not just SQL Server, it’s not just Analysis Services. Is it SQL DB in Azure, is it Managed Instances, is it SQL DW, is it SQL in a VM, is it Data Lake, is it Databricks, is it R, is it Python, is it Data Factory, is it Integration Services, is it Power BI, is it Flow\u0026hellip;\nKN: Even not counting the cloud solutions, in the package of SQL Server on premise you have now PolyBase, the R language, Columnstore, all these technologies that you need to learn.\nAW: I mean it’s great that all this technology exists, but it makes it actually quite difficult to be an expert in everything. And so knowing what you want to focus in is actually a really big challenge. So anyone starting out in this industry, if you’re trying to choose an area to go into, first of all, the Microsoft Data Platform world is a wonderful place to be in. Compare it to other competing environments. Just the community itself is so much more sharing open and welcoming, you will find it much easier to learn and find it easier to make contacts in the Microsoft world than other worlds, which is why I still work in a Microsoft world. By coming to events like SQL Relay, by going to SQLBits, SQL Saturday, all these community events, it gives you an opportunity to spend an hour learning about a particular topic. Now you may not leave that hour being an expert in it, but you’ll understand the concept of what it is. And so it’s getting more and more important to be aware of all the different components in the Data Platform. Not to be an expert in every one of them, but to be aware that they exist and to know, how they may fit into a jigsaw, so that you can focus on the area that you’re currently working on, but when you get a problem, it’s having an awareness of what other technology is available to you to be able to solve that problem. And you can think back: “OK, last year I saw a talk on, I don’t know, Flow for example. Great, I’ve seen that work in a demo, I think that may be suitable”. I’ll now spend a few days looking into it, playing with it and actually trying it out. But unless you’ve seen a talk at a conference or read a blog, how would you know Flow even exists? How would you know whether Databricks or Data Lake are valid options for a big data analytics solution? So it’s important to have an overview of all these different technologies, so if you do go to the conference, don’t just go to the sessions that tick the box of what you’re working on now. Broaden your horizon, go and see the sessions that don’t tick your boxes, the things you don’t know you need to know about and you will find out, that a number of those topics actually do become very, very useful for you in future months or years. Because at some point you’ll hit a problem and you’ll think back to that session and think “Yeah, I know that technology is gonna work”, but also you know who the speaker was. You know if you need help with it. You know an expert you can ask. Most of our speakers are, if they’re not consultants or contractors, they’re very helpful people and they like talking about this technology. Not a single one of them wouldn’t respond to an email asking for some help or advice. So it’s a very good way of finding out useful information about other topics. So that means you’ve got a wide breadth of awareness of tools and that then means, whatever job you go and look for, you can start to expand your breadth of knowledge, and over time focus. Not once in my career did I ever decide to be a business intelligence consultant, I kind of fell into it through a series of opportunities, mistakes, scenarios that I didn’t plan or expect. And I’ve ended up doing this job that I absolutely love with a passion. But I never chose to do it. I just happened to end up doing this through a sequence of random events. So don’t ever block off channels or don’t close your mind to opportunities. If there’s an opportunity to get into a particular technology, take it with both hands and the community will help you learn.\nKN: So you also have your own blog, yes?\nAW: I do have a blog. Having said that, it may not be as up-to-date as it used to be. I used to religiously blog at least once a month. These were always quite in-depth technical blogs that may take a number of days to write. Sometimes even weeks to write, days or weeks, yeah. So some blogs I was preparing for even months of refining code and getting it right and getting it ready and then it’s ready for a blog. I would use the blogs as a way of actually starting out with ideas about problems I’d solved or different interesting techniques of data. And then, if the blog post took off and gained traction with comments and views, then I would turn that into a conference talk and I would go and present that in SQL Saturday, user groups, SQLBits, PASS Summit, etc. Then, as I started to get more involved in actually running the community, so now I run the Birmingham Data Platform meet ups, I’m director of SQL Relay, I’m also on the committee for SQLBits. Those three combined, especially Data Relay and SQLBits take up a phenomenal amount of my time and they pretty much take up all the time that I used to have to write blogs. So I think I’ve only written two blog posts in the last six months. I do need to get back to that, because I find that a really good way of learning myself. If you want to learn new technology, set yourself a challenge, set yourself a scenario to solve, figure out a solution to it, write a blog about it. It’s gonna help someone else. It also helps crystallize in your own mind, how a technology works. Doing a conference talk really helps you learn a new technology when you’re preparing the talk. But at the moment, I tend to focus on doing talks at conferences rather than writing blogs.\nKN: Completely understandable. I think you shouldn’t worry about it because during my session I was leading two days ago at Birmingham, the second day, sorry, Leeds. When I mentioned SCDs, one of the attendees mentioned your blog post. Most probably we read the same posts. I still remember I saw a lot of comments under that post.\nAW: I did a series of blog posts, I think three blog posts on Slowly Changing Dimensions in Data Warehouses and how to use the T-SQL MERGE statements to automate that. To this day that is by a country mile the most popular blog post that I have ever written, or series of blog posts. I actually did my master’s degree thesis on that topic, on the performance characteristics of loading SCDs and the blog posts came out as a result of that. Actually, my second ever SQLBits talk was on that topic. What’s really great, what I love is I wrote that blog post about eight years ago, something like that, and still to this day I get comments on “is this still the best approach to doing it?” Well, I had that comment last week.\nKN: The MERGE?\nAW: Yes, the MERGE. Absolutely. I use it for cloud solutions, on-premise solutions, every single day.\nKN: Have you heard about SCD Merge Wizard? It’s an application that helps you create a MERGE statement for mapping your source and target. Then you can define the business logic behind it using type 1, 2, 3 and 6.\nAW: I have not come across that.\nKN: I’m a member of this project, I became a member of this project like 2 or 3 years ago, because one of the original author didn’t have time to develop that project, so I was happy to continue that work.\nAW: What’s great about that is it shows the concept of community helping each other, so when I first did that post, using MERGE to do that was never considered an option, no-one had really done it before, but to me it seemed a perfect opportunity to use new functionality that was being introduced in the SQL Server engine to solve a complex problem in BI loading. And when you’re trying to manage SCD loading and Integration Services, it’s not a pretty thing, it doesn’t perform very well. So this to me seemed a perfect marriage of these concepts so I spent a lot of time investigating and researching the process of doing that and it works brilliantly. That concept of using MERGE is now completely taken off and everywhere I go, I speak to people who were using MERGE to load their data warehouses and then other people, like you, the same with your project, you’re then enhancing that and taking it forward, improving it again. I did a blog post on series of talks, my first ever SQLBits talk was about automating the documentation of cubes. No one likes writing documentation, it’s a boring process, so I wrote a process of using DMVs, Dynamic Management Views within SQL Server Analysis Services to query the metadata and structure of a cube, pull that into Reporting Services and then visualize the structure of a cube, basically creating automated real-time documentation, purely to save me time writing documentation. I give that to every customer and I put all the code on a blog and that was probably nine years ago, something like that. And it still works! But that’s Multidimensional Cubes. I was talking to Steve Powell, another community leader in the data platform. And he’s taken that code off the blog, converted it, enhanced it, improved it, added some PowerShell to make it work against Power BI, so you can now use that similar concept, but enhancing it another stage further, so he’s taken that code, built on it, I’m now taking his code and building it further, and so we’re all helping each other out and we all gain better code and we get more features and functionality. And when you start sharing stuff, you get it back again, everyone gains from it, and that’s what I love about this community.\nKN: That is the perfect idea behind the GitHub. Open source code.\nAW: Yeah.\nKN: So we are talking about the communities, SQL family members, etc. What do you think about the MVP program these days?\nAW: It is fantastic. I am very, very fortunate to have been awarded the MVP award by Microsoft. I’m a passionate supporter of the program, both from the value to the community, it gives you a level of contact within Microsoft. Obviously, there’s very strict non-disclosure agreements that we have to sign with Microsoft, so we won’t disclose any secret information outside of the program, but that means that the designers, architects and program managers within Microsoft can then share information with us about the future direction of their platforms – roadmaps. And that does two things: first of all, it means we, as a community, are able to feedback to Microsoft and say what we love about that, what we’re a bit concerned about, where we think that could be tweaked, enhanced, and provide very early feedback on the pros, cons and benefits of their approach. It also means we can become familiar with the concept and the approach to what’s happening and start writing blogs, conference talks.\nKN: Sharing the knowledge, yes?\nAW: Keeping that private initially, but have content ready, so that as soon as Microsoft release a bit of functionality, we can hit Publish on our blog post and straight away other people in the community want to learn about that, there’s already content for them to go and look at and learn. And as soon as you get announcements at Microsoft Ignite, places like that, you will straight away see a lot of blog posts, a lot of people talking on Twitter about the topics, because we’ve been given the opportunity to actually have inside information to see that beforehand. And I love that, because it gives us an opportunity to actually get a head start and provide more content to the community. What it also means, from a consultant perspective, is that when I’m designing architectures and designing solutions for a customer, I can’t tell them why I’m making decisions, but I can make certain decisions on choosing one architecture over another based on decisions that I know are going to be clear to them in six months’ time, but not now. I’ve got two customers at the moment who want to go down a particular route with the BI solution that we have for them, and I’m specifically holding off on that for a few months, because I’m aware of an announcement that’s going to be made on SQLBits in February that will change what they can do with that solution. And I know we will make their solutions easier, more simple, faster and better. And so it allows me to help make better decisions on behalf of my customers. And so from that perspective, it’s very valuable. From the consulting perspective, what it does mean is we also get access to play around with a lot of Microsoft tools. So obviously when you’re playing with an on-prem SQL Server instance there’s no cost involved. You’ve already got that environment there. When you’re playing with Azure, Azure is not free. All the services charge by the hour or by the instance you have. Data Warehouse and Hosted Analysis Services, if you scale those up, then they can cost some decent amount of money. And so when we’re trying to put together conference talks or blog posts or really test the scalability and performance characteristics of these tools, that can be very expensive. So as an MVP Microsoft give us, not unlimited unfortunately, but a reasonable amount of Azure credits that we can use to play with this technology to really get to grips with it, which means we can then pass that on to other people. And I think probably the most valuable part of the MVP year is the MVP summit in Seattle. An opportunity where all MVPs around the world and every group, not just Data Platform, get invited over to Redmond and we spend a week with Microsoft, with the team leaders, the program managers, the architects, the developers and they talk us through the roadmaps. But also we get to have a lot of one-to-one conversations with the teams about technology, give them feedback, get an understanding from them as to why decisions have been made and the reasons for things. And that helps us then explain to the rest of the community how things can work. So it’s an incredibly valuable program, and I’m thrilled to be a part of it.\nKN: Sounds very good. And you, as a speaker, how do you prepare yourself for a speech?\nAW: That’s changed over the years. I always used to base a talk off blog posts, so I’d write a blog post and then evolve that and turn that into a talk. As my time blogging has decreased, I tend to just jump straight in and do a talk now. It’s still based on identifying things that I enjoy talking about and technologies that I find fascinating or interesting. That may be relevant to a particular project that I’m working on at the time, so at the moment we’re doing a lot of Analysis Services in Azure migrations. Migrations from Multidimensional Cubes to Tabular Cubes, Power BI to Tabular Cubes. So I’m doing a number of talks about what Analysis Services in Azure can do, how to manage it, scalability, etc. Because I’m investigating that for my customers anyway. I’m doing that as part of my job and everything that I learn about it is, obviously hopefully, useful for someone else. I can explore different options and the ways of managing it and then share that with other people. I did a talk this morning in the keynote here about designing a deep learning neural network to play games. Now, there’s no business reason why I’m teaching a neural network to play games, that’s just pure fun, but in my world as a BI consultant, it used to be all about getting reliable, clean, accurate data into a Data Warehouse and present it with self-service reporting. That’s still very much a core, fundamental part of BI, but once you’ve got that solid foundation of data, these days machine learning algorithms are so easy and accessible, whether that’s through SQL Server, R, or Python code, whether that’s through Databricks or Data Lake or R-studio, doesn’t matter, it’s accessible. And it becomes very easy to start doing customer churn predictions or time series modeling or future predictions or basket analysis. All these kinds of machine learning algorithms are very accessible now.\nKN: Very accessible but still I think, I feel that very new for most of the T-SQL developers or BI developers.\nAW: Absolutely, and that’s exactly why I did that demo in the keynote today. It was to show, with a hundred and eighty lines of Python code, you can create a deep learning neural network that will learn how to play a game with no guidance. And it works. And so ten years ago, five years ago you had to be a data science expert to write that kind of code. Now, with tools like Cognitive Services, with TensorFlow, tools like that, it’s actually much more accessible, so BI people, like me, DBAs don’t need to go and do a PhD to understand data science. They can write code now. OK, to do a really complex model, I’m not belittling the value of a data science professional, but you can start out playing with this tech to understand this in a really easy way. So as my day job is getting more and more involved in data science, I’m learning myself how different tools work, such as TensorFlow. So I wrote that game or started playing with that game in order to learn myself better. And I just really enjoyed that demo, so I thought it would be useful to share. To share with people that it isn’t as scary as it may seem, so a lot of my talks actually stem from that now. When I’m learning a new technology, I’ll find something that I think is valuable and I’ll just turn it into a talk and evolve it. And then every time I go and do a talk, I get feedback on it and I’ll extend it, enhance it, based on the feedback and it evolves over time.\nKN: Can you show that code, attached to this post?\nAW: There’s actually a lot of sample code already out there that does that. So actually I started by working through a YouTube tutorial with a guy, I can’t remember the name of the chap, but if you search for Open AI Gym and Cart Pole on YouTube, you will find a four-post series of videos where this guy talks through a step-by-step process on all the Python code to use this and to write that game. And in the Open AI Gym library there’s a number of games you can play and you can modify his code to do various things. So that’s actually how I started playing around with this code and then evolved the code, and developed it myself to do different things. So there is plenty of code and tutorials already there. It’s just I don’t think people are looking for it because they think it’s really complicated and really out of their limit, but I wanted to show that actually anybody can jump in and spend a couple of hours playing with it.\n**KN:**That’s why I would like to attach your code, because you showed this small game based on neural networks?\nAW: Deep learning neural networks. What I’ll do is I’ll give you a link to this YouTube series. That talks it through in a really clear way with pretty much the same result as I was getting. So I’ll give you a link to that and then give that guy some credits for training me on how to do it!\nKN: Tell me about your work-life balance.\nAW: If you ask me or ask my wife you may get different answers. It’s something that I always struggle with. I passionately enjoy my job and I run my own business. Anyone that runs their own business knows, there’s no such thing as a day off. You’re always on duty. I started out as a freelance consultant and you’re working seven days a week, building a new business. I’m quite fortunate now to have a team of six and I’ve got a team that can carry on doing the work without me in the business, although it’s still very difficult to detach myself from it. About two years ago, I had to make a life shift, where I was working seven days a week. I was working 12-14 hours a day in the office and I had a discussion with my wife Hollie who is the biggest supporter and the biggest influence in my life. Every decision I make that’s been good in my life has basically come from Hollie. So I owe her a lot, she’s my biggest supporter, pusher and mentor, I suppose. And it was a case of I need to slow down a little bit of work and find a hobby outside of work to try and detach myself from being in the office all the time. So I took up flying. I wanted something that forced me to get away from work and to think about something else. My dad used to fly when he was in his twenties and I’ve got a couple of very good friends who are pilots so they encouraged me to take up flying. So I started learning. I got my pilot’s license after, I think, about six months of training and yeah, I’ve been flying ever since. And I absolutely adore it. It’s a total detachment from reality and you have to focus and concentrate so hard on everything that’s going on, that I find it incredibly relaxing because it forces me to stop thinking about work. You’re focusing on radio, you’re focusing on actually flying the plane, on navigation, on monitoring, on fuel management\u0026hellip;\nKN: There’s autopilot!\nAW: Oh, that’s only for jet pilots. Yes, I’ve got autopilot on the plane, but I never use it. There’s no fun, is it? You actually want to fly the thing. So I very rarely turn it on. I prefer to fly myself. But I find it’s like juggling ten balls, there’s so much to think about that you can’t worry about work, you can’t worry about everything else that’s going on in life. It forces you to detach. But it’s also an incredibly social thing. I go flying with my friends, we’ll fly off on Sunday morning for a bacon sandwich somewhere the other side of the country. To drive down to Cornwall takes about three and a half to four hours. I flew there with my sister the other week for an ice cream. We flew down there, had an ice cream on the beach and flew back again.\nKN: I’ve seen that you also fly to the customers?\nAW: Actually, one decision-making factor in which customers we take on may be related to how close they are to an airfield. It’s not a serious consideration, but if there is an airfield nearby, then that’s nice to be able to fly to a customer. You can’t always manage it with weather, but it’s good fun.\nKN: So you are a celebrity.\nAW: I wouldn’t quite say that.\nKN: Wait a minute: local newspaper has written about you in a Spitfire. Let me quote: “Alex swaps digital cloud for the real thing – a Spitfire treat”\nAW: Yes, that was a fun day. It was my birthday weekend and my wife Hollie, she’s a musician, so she plays clarinet for the Birmingham Philharmonic Orchestra, a very talented musician. And so because she’s out every Sunday rehearsing, I had this day free on the Sunday and I thought I’d go flying and I was looking at where to go on the day and I spoke to a couple of friends.\nAlex Whittles takes to the air in the Spitfire. Source: Shropshire live My friend John was free, he’s a pilot as well, and so we thought, we’d go for a jolly for the day. We’re always looking around as to where to go and I’ve never been to Duxford and there’s obviously the air museum there. And John suggested that we go there, so I flew us down to Duxford. It took an hour and ten minutes, something like that, it would have taken three hours in the car. And as we were coming into land, the air traffic control came on the radio and basically said “I’m sorry, we’re giving priority to a Spitfire coming in”. You could always get priority to a Spitfire. If you’re flying a Spitfire, you own the skies, you can have the runway. And so the Spitfire came in in front of us and we had to do a last-minute diversion off onto the grass runway parallel to it. And so we landed parallel with this Spitfire, just behind the Spitfire. It was just a magical experience. So just flying close to the Spitfire, seeing it in the air, it was just something else. And so we landed and then parked up the plane and then wanted to go and have a look at the Spitfire and have a chat with the pilot, get a photograph. And it turns out that they sell flights with a Spitfire from Duxford, which I didn’t realise. And they had one flight that was free, it was available on that day. It was my birthday weekend so I thought “once in a lifetime opportunity”. I can’t really justify the cost of it, but how often do you get an opportunity to fly a Spitfire. And it turns out, my friend who went down with me, John, about six or seven years ago had flown that same Spitfire, which used to be owned by Red Dragon up in North Wales and he’d flown it up there. And so he was the little devil on my shoulder saying “do it, do it, do it” and so I signed up to do the flight. And also because I’ve got my pilot’s license, I was able to take control of it and we had a half-hour flight around Duxford, around Cambridgeshire, we did a victory roll in it and then he gave me the controls. We were flying in close formation with a Harvard, with my friend John in the Harvard, so I’ve got the most fantastic photograph from the Harvard looking at me in the back of this Spitfire. And to this day it’s just my favorite photograph that’s ever been taken. And that experience of actually feeling the raw power in that Merlin engine and the sensitivity of the controls, compared to the PA-28 that I normally fly, which is like flying a Cadillac, very relaxed and forgiving. And with a Spitfire you so much just touch the controls and you’re flung sideways. The sensitivity and that power is just\u0026hellip; you cannot imagine this until you get into the seat. What really struck me was whenever you hear a Spitfire, it’s an unmistakable sound of that Merlin engine. It’s instantly recognizable, that’s a phenomenal sound. Once you get into the Spitfire – completely different. It doesn’t sound anything like it, it’s really strange. Fantastic experience. The only slight issue was it was a little bit of a squeeze to get into the cockpit. I’m 6′6″, Spitfires weren’t really designed for people of my stature, so you can look at the photograph and see my head actually just touching the roof of the cockpit, squeezing into the glass, so it was a tight squeeze, but it was an incredible experience. I actually flew back into Duxford again about three weeks ago and had exactly the same experience of flying in formation with the Spitfire as we were going around the circuit coming into the land and again, had to divert off to the grass strip again next to the Spitfire, but it was lovely. It was such an incredible experience.\nKN: So sounds like you found a perfect hobby.\nAW: Absolutely, I love it. And it gets me away from work. And it really does help the work-life balance. They say a change is as good as a rest. Taking your mind away from something. Now whether that’s flying or for my wife it’s music, it’s going to see her doing concerts, anything like that that takes your mind away from whatever is stressing you out on the day. The day-to-day grind of normal, day-to-day life, we all have it, we all have stress. It’s finding something that detaches you from that and gives your brain a break to think about something else, and you get back more refreshed. And I find having that little bit of time away whenever I go and fly every couple of weeks, just gives me a renewed vigor for work when I’m going back again. So I’m learning how that work-life balance needs to work at the moment, and it’s an evolving process but I’m trying.\nKN: Like all of us! The last question. Children?\nAW: We don’t have children. I have eight delightful godchildren and two nieces and two nephews. And I’m very fortunate, I get to spend a lot of time with them and they’re all wonderful. The beauty of them is that when they play up, you can give them back to their parents.\nKN: Alex, at the end of our conversation, tell us where we can find you.\nAW: Where you can find me? I’m on Twitter with @PurpleFrogAlex. My website is www.purplefrogsystems.com. You can find me on my blog, it’s www.purplefrogsystems.com/blog/. Or if you go onto the Contact Us on the SQLBits website, those emails come straight to me. Or you can find me at most events around the UK or around Europe speaking or helping.\nKN: Thank you very much for this conversation. Thank you very much indeed.\nAW: It’s been a delight speaking to you, thank you very much.\nUseful links Alex Twitter: @PurpleFrogAlex Alex\u0026rsquo;s websites: Purple Frog System (company) | BLOG AI articles/videos: Deep learning neural networks | the Open AI Gym library | TensorFlow Article: Alex swaps digital cloud for the real thing – a Spitfire treat Alex\u0026rsquo;s post about SCD \u0026amp; MERGE: Using T-SQL Merge to load Data Warehouse dimensions Conferences: SQLBits, Data Relay (formerly SQL Relay)\n","date":"2018-12-14T06:00:58Z","image":"/uploads/2018/12/ASF017-DataRelay-with-Alex-Whittles.jpg","permalink":"/2018/12/asf-017-alex-whittles-interview/","title":"ASF 017: Alex Whittles interview"},{"content":"It is a first post on series related to installation and configuration of SQL Server installations.\nThis series is for professionals who starts their journey with SQL Server administration and also for those who want to extend and structure their knowledge on SQL Server administration.\nPreparations We are starting with following assumptions:\nIt is brand new installation on standalone server We are using Windows Server as an operating system Server is up to date with all security updates from Windows Update Database server is a part of Active Directory domain Active Directory schema is at least the Windows Server 2012 domain functional level All drives used for SQL Server are configured as independent LUNs Latest publicly available version of SQL Server will be used. As a first step we need to check if all prerequisites for the installation are met. Most important is to have .NET framework installed and server is restarted before installation.\nFor installation of .NET framework, we need to have ISO file of the Windows Server.\nHow to install .NET Framework in Windows Server environment? As a first step start Server Manager and go to Manage -\u0026gt; Add Roles and Features\nOn Before you begin screen click Next\nOn Select installation type screen select Role-based or feature-based installation and click Next\nOn Select destination server choose Select a server from server pool and select the server and click Next\nOn Select server roles click Next\nOn Select features screen expand .NET Framework 3.5 Features and select .NET Framework 3.5 (includes .NET 2.0 and 3.0) and click Next\nClick on Install to proceed with installation.\nYou can also check if .NET is installed using PowerShell command:\nGet-WindowsFeature -Name Net-Framework-Core Configuration of the disks for SQL Server Once .NET is installed we can proceed to configuration of separate drives for usage by SQL Server - it is common practice due to mainly performance advantages of this setup.\nIn my lab environment, I will be using following drives:\nD:\\ for binaries of SQL Server and system databases 3 drives for Data files of user databases configured as mount points on D:\\ drive 1 drive for Transaction Log of user databases configured as mount points on D:\\ drive 4 drives for tempdb configured as mount points on D:\\ drive - in my lab I am using virtual machine with 4 cores thus 4 separate disks should be used. As a starting point, you should start with 1 tempdb data file per CPU core up to 8 files 1 drive for backups configured as mount points on D:\\ drive Before we start with installation, it is required to format drives with desired settings. You may wonder what can be configured with configuration? Yes, there are some settings that should be adjusted for drives used by SQL Server:\nFile system - SQL Server support NTFS and ReFS Allocation unit - should be changed from default (512 bytes) set to 64k for drives used for data Deselect \u0026ldquo;Perform quick format\u0026rdquo; option - this option will force to format physical storage, avoid dynamic allocation. This will minimize random performance hiccups during normal operation. As a first drive will be used, I will configure it as a normal drive with D:\\ drive letter. Below step-by-step screen shots how to configure the drive.\nBrowse to Computer Management (Local) -\u0026gt; Storage -\u0026gt; Disk Management and right-click on first disk\nSelect Online option\nOnce disk is switched from Offline state to Not Initialized right-click and select Initialize Disk\nOn Initialize Disk pop-up select all disks and use GPT (GUID Partition Table) option\nDisk should change it\u0026rsquo;s state to Online\nRight-click Unallocated space and select New Simple Volume\u0026hellip;\nOn the New Simple Volume Wizard click Next\nOn the Specify Volume Size provide size of the volume and click Next\nOn the Assign Drive Letter or Path select D drive and click Next\nOn Format Partition select NTFS file system and correct allocation unit - for all drives used for storing users data files, allocation unit should be set to 64K. Here we are formatting drive for binaries of SQL Server thus using default 512. Deselect Perform a quick format option.\nOn the summary screen click on Finish to start formatting of the drive.\nNext drives will be configured as a mount points on the D:\\ drive. I am using mount points due to fact that it is not commonly used and can help us with having too many drives in the system (situation when all drives letters are used).\nBelow step-by-step screen shots how to configure the drive as a mount point.\nRight click on Disk 2 and select Online\nOn Initialize Disk pop-up select all disks and use GPT (GUID Partition Table) option\nRight-click Unallocated space and select New Simple Volume\u0026hellip;\nOn the Specify Volume Size provide size of the volume and click Next\nOn the Assign Drive Letter or Path select Mount in the following empty NTFS folder and click Browse\u0026hellip;\nSelect D:\\ drive and click on New Folder\u0026hellip; to create folder called SQLData00. Select this folder and click OK\nOn Format Partition select NTFS file system and 64K *allocation unit,*provide Volume Label (SQLData00) and Deselect Perform a quick format option.\nOn the summary screen click on Finish to start formatting of the drive.\nRepeat the same steps for the other drives (SQLData01-02, SQLTLog, SQLTempDB01-04 and SQLBackup). Once all steps are completed you should see following drives in This PC:\nWhen browsing to D:\\ drive all mount points should be visible and in online state.\nGroup Managed Service Accounts for SQL Server One of the new feature of the Windows Server 2008 R2 was Managed Service Account (MSA). This feature allows you to create an Active Directory account that is tied to specific computer object with complex password and automatic process of password renewal (like for the computer object). Such object significantly increase security as the Database Administrator (DBA) doesn\u0026rsquo;t need to change password for SQL Server accounts. One of the restriction of the MSA was that it can\u0026rsquo;t be used for more than one computer. This limitation was solved new feature in Windows Server 2012 called Group Managed Service Account (gMSA). Below there are steps describing how enable gMSA on Active Directory and how to configure SQL Server to take advantage of it.\nAs a first step Key Distribution Service (KDC) Root Key needs to be created. For that we are going to execute PowerShell command on domain controller by user with Domain Administrator or Enterprise Administrator privileges.\nAs we don\u0026rsquo;t want to wait default 10 hours, additional parameter -EffectiveTime has been used:\nAdd-KdsRootKey –EffectiveTime ((get-date).addhours(-10)) Second step is to created Global Security Group for all Computers where SQL Server will be installed\nNew-ADGroup -Name \u0026#34;SQLServers\u0026#34; `\r-GroupCategory Security `\r-GroupScope Global `\r-DisplayName \u0026#34;Production SQL Servers\u0026#34; `\r-Path \u0026#34;CN=Users,DC=datacommunity,DC=local\u0026#34; `\r-Description \u0026#34;Group for all SQL Servers using gMSA\u0026#34; Third step is to add all SQL Server computers to the group created in previous step:\nAdd-ADGroupMember \u0026#34;SQLServers\u0026#34; –Members “SQL$” To validate memebers of the AD Group, following command can be used:\nGet-ADGroupMember \u0026#34;SQLServers\u0026#34; Fourth step is to create gMSA accounts for all services that we are going to use - in our case two accounts:\nSQL Server Database Engine SQL Server Agent To add accounts following PowerShell command can be executed:\nNew-ADServiceAccount -name SQLServerEngine `\r-DNSHostName SQLServerEngine.contoso.com `\r-PrincipalsAllowedToRetrieveManagedPassword SQLServers\rNew-ADServiceAccount -name SQLServerAgent `\r-DNSHostName SQLServerAgent.contoso.com `\r-PrincipalsAllowedToRetrieveManagedPassword SQLServers On the member server (SQL computer in our case) there are additional steps that needs to be executed:\nImport AD module by running:\nAdd-WindowsFeature RSAT-AD-PowerShell Both gMSA accounts needs to be installed on the member server by running following command:\nInstall-ADServiceAccount SQLServerEngine Install-ADServiceAccount SQLServerAgent To check if gMSA are installed correctly you can run Test-ADServiceAccount cmdlet:\nTest-ADServiceAccount SQLServerEngine\rTest-ADServiceAccount SQLServerAgent After this step - Active Directory domain and member server are ready for the installation of SQL Server.\nSummary Proper configuration of the Active Directory (AD) domain and operating system can give significant performance, security and manageability advantages. Every time when you build new server for SQL Server, check if all steps described in this blog post are configured.\nLinks More information about described above steps can be found on following sites:\nhttps://docs.microsoft.com/en-us/dotnet/framework/install/on-windows-10\nhttps://docs.microsoft.com/en-us/windows-server/storage/disk-management/overview-of-disk-management\nhttps://docs.microsoft.com/en-us/windows-server/security/group-managed-service-accounts/group-managed-service-accounts-overview\nhttps://docs.microsoft.com/en-us/previous-versions/sql/sql-server-2008/dd758814(v=sql.100)\n","date":"2018-12-12T07:21:52Z","image":"/uploads/2018/12/athletic-field-1867053_1280.jpg","permalink":"/2018/12/preparation-for-sql-server-installation/","title":"Preparation for SQL Server installation"},{"content":"Good day SQL folks!\nHere is your weekly list of articles worth to read!\nAzure Cosmos DB SQL language reference\nAzure Cosmos DB supports querying documents using a familiar SQL (Structured Query Language) like grammar over hierarchical JSON documents without requiring explicit schema or creation of secondary indexes. This article provides documentation for the SQL query language syntax, which is compatible with SQL API accounts.\nAzure Cosmos DB partitioning design patterns – Part 1\nIn this article, you will learn how to use partition keys to efficiently distribute data, improve application performance, and enable faster look-up.\nAzure SQL Managed Instance Business Critical tier is Generally Available\nHigh availability is implemented using SQL Server Always On technology that ensures that changes made on the primary server are replicated to at least two replicas before the transaction is committed.\nManage consistency levels in Azure Cosmos DB\nThis article explains how to manage consistency levels in Azure Cosmos DB. You learn how to configure the default consistency level, override the default consistency, manually manage session tokens, and understand the Probabilistically Bounded Staleness (PBS) metric.\nCloud Application Architecture Guide\nThis cloud computing architecture e-book focuses on architecture, design, and implementation—considerations that apply no matter which cloud platform you choose\nCheers,\nDamian\n","date":"2018-12-09T23:54:15Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/12/last-week-reading-2018-12-09/","title":"Last week reading (2018-12-09)"},{"content":"\nWhat wasn\u0026rsquo;t really designed for people of guest\u0026rsquo;s stature? From who his every good decision does come from?\nNew podcast episode is coming (next Friday) and in the meantime you can watch trailer.\nSimply click PLAY button above or move to the YouTube: https://youtu.be/hbs01wbEcHI\nEnjoy!\n","date":"2018-12-07T07:00:00Z","image":"/uploads/2018/12/ASF-017-video-invitation.jpg","permalink":"/2018/12/asf-017-video-invitation/","title":"ASF 017 video invitation"},{"content":"In coming weekend I will be speaking at SQL Saturday in Ljubljana, Slovenia.\nI\u0026rsquo;m extra happy because I do it again, third time in a row. If you\u0026rsquo;ve been there you already know that it special event.\nNot only because its the last SQL Saturday in a year in Europe, but also the event is leading by four great guys: Dejan Sarka (MVP), Mladen Prajdic (MVP), Tomaz Kastrun (MVP), Matija Lah.\nAzure Data Factory v2 with Data Flows capabilities This time, I will be talking about Azure Data Factory and its new feature: Data Flow, which is still in preview. I gonna show how to create ADF briefly and then focus mainly on Data Flow - how it works, when it\u0026rsquo;s useful and why it might be so powerful. Please come and see me presenting this topic if you\u0026rsquo;re interested in.\nAgenda As always, SQL Saturday events are free, so feel welcomed.\nThe event in Ljubljana has 30 sessions across 5 tracks: BI, BI/DEV, DEV, DBA, DEV/DBA.\nClick here if you wanna check full agenda of Saturday.\nMemories It is been almost a year now since last event in Slovenia and very busy time for me, so I did not publish pictures from that event. This is good moment to share the pictures from last year\u0026rsquo; event.\nBecause I want to keep events chronologically - I put that post, back in time.\nClick here too see all the pictures from SQL Saturday #687 Ljubljana 2017.\nAnd hey! Look who I was sitting next to, on the way back. That was enjoyable coincidence as I was not reserve my seat in advance.\nPrathy, are we gonna have the same flight, this year? ","date":"2018-12-06T11:20:53Z","image":"/uploads/2018/12/img_5c070f1751a72.png","permalink":"/2018/12/speaking-at-sql-saturday-782-slovenia/","title":"Speaking at SQL Saturday #782 Slovenia"},{"content":"Many of you (including me) wonder about it.\nNamely: Is it possible to move my ETL process from SSIS to ADF? How can I reflect current SSIS Data Flow business logic in Azure Data Factory? And turned out that new feature in ADF: Data Flow - comes with help. Furthermore, such solution will be scalable as Azure Databricks works under the hood. Don\u0026rsquo;t worry - you don\u0026rsquo;t have to know Databricks and extra language (Scala, Python) at all. How does it possible? Carry on reading.\nIn this post\u0026rsquo;s section, I would like to show you what kind of actions you can do and what is their equivalent in SQL and SSIS.\nNew world: Data Flow in Azure Data Factory The big benefit here is that you will not write any line of code. You can design whole business logic from the scratch using Data Flow UX and appropriate code in Scala will be prepared, compile and execute in Azure Databricks behind the scenes. So that you can focus on business logic and data transformations like data cleaning, aggregation, data preparation and build code-free dataflow pipelines.\nAdditionally, the process would be automatically scale-out if you allow for that.\nADF Data Flow vs SSIS vs T-SQL The main purpose of this post is to bring capabilities of (ADF) Data Flow closer and compare it to its counterparts from SSIS and relevant code of T-SQL.\nWhy? Because it\u0026rsquo;s far easier to understand something new by comparison to something that we know very well.\nFurthermore, tables and icons talks to us much more, hence it is easy to acquire such new knowledge.\nHaving those fundamentals, you can re-design the current ETL process in Azure Data Factory when having a clear image of mapping components between SSIS and ADFDF. To fulfil the picture out, I have added a column that shows T-SQL code that does the same or similar things in SQL.\nSo, no matter which technology your current process uses, either Stored Procedures in SQL or SSIS, you are able to sit down and recreate that process to uncover new opportunities.\nComponents Operation / Activity Description SSIS equivalent SQL Server equivalent New branch Create a new flow branch with the same data Multicast (+icon) ``` SELECT INTO SELECT OUTPUT ``` Join Join data from two streams based on a condition Merge join ``` INNER/LEFT/RIGHT JOIN, CROSS/FULL OUTER JOIN ``` Conditional Split Route data into different streams based on conditions Conditional Split ``` SELECT INTO WHERE condition1 SELECT INTO WHERE condition2 CASE \u0026hellip; WHEN ``` Union Collect data from multiple streams Union All ``` SELECT col1a UNION (ALL) SELECT col1b ``` Lookup Lookup additional data from another stream Lookup Subselect, function, LEFT/RIGHT JOIN Derived Column Compute new columns based on the existing once Derived Column SELECT Column1 * 1.09 as NewColumn Aggregate Calculate aggregation on the stream Aggregate ``` SELECT Year(DateOfBirth) as YearOnly, MIN(), MAX(), AVG() GROUP BY Year(DateOfBirth) ``` Surrogate Key Add a surrogate key column to output stream from a specific value Script Component ``` SELECT ROW_NUMBER() OVER(ORDER BY name ASC) AS Row#, name FROM sys.databases ``` *Incremental Primary Key (with limited capabilities)* Exists Check the existence of data in another stream Lookup / Merge Join ``` SELECT * FROM Table WHERE EXISTS(SELECT \u0026hellip;) ``` Select Choose columns to flow to the next stream OUTPUT in components, mapping columns ``` SELECT Column1, Column4 FROM Table ``` Filter Filter rows in the stream based on a condition Conditional Split ``` SELECT * FROM Table WHERE [Column] LIKE \u0026lsquo;%pattern%\u0026rsquo; ``` Sort Order data in the stream based on column(s) Sort ``` SELECT * FROM Table ORDER BY [Column] ASC ``` Extend Use any custom logic from an external library Script Component SQL CLR Source Source for your data flow. Obligatory first element of every Data Flow in ADF. OLE DB Source and more … SELECT * FROM SourceTable Sink Destination for your data flow OLE DB Destination and more… INSERT INTO TargetTable Update 04/01/2022\nDo you think the above table is useful? Download the updated version (PDF) as a two-page cheat sheet.\nMore interesting materials like this can be found in the following free course: Cheat sheets for Data Engineers\nSummary This new feature has huge capabilities. I\u0026rsquo;m very excited being had opportunity to use it more.\nAn automatically scalable process, like this, might be very efficient with Big Data processing. Hence, it\u0026rsquo;s worth to start designing new processes with Azure Data Factory or even migrating existing processes when your enterprise suffers from performance degradation due to the amount of processing data.\nPlease be aware that among Microsoft solutions is another Data Flow - exists in Power BI. Do not confuse them.\nIn next posts of this series, I will be explaining all activities from ADF Data Flow a bit deeper.\nLet me know your thoughts or leave a comment once you have any questions.\nThanks for reading!\nUseful links ADF Data Flow\u0026rsquo;s documentation\nADF Data Flow\u0026rsquo;s videos\nFollow this tag on the blog: ADFDF\n","date":"2018-12-05T01:21:40Z","image":"/uploads/2018/12/lapadu-1385450_1920.jpg","permalink":"/2018/12/azure-data-factory-v2-and-its-available-components-in-data-flows/","title":"Azure Data Factory v2 and its available components in Data Flows"},{"content":"Hello folks, This is a quick roundup of Internet posts from last week. Have a good reading!\nCreate and configure a self-hosted integration runtime\nThis solution allows you integrate your Azure environments with on-premises (or VM) machines.\nUnderstanding log buffer flushes\nItzik Ben-Gan (T) explains what techniques worth to use when an OLTP-style workloads require high frequency and low latency.\nHow to handle SSRS multi-value parameter filtering in SQL Server Parallel Data Warehouse\nBypass all the limitations of PDW or Azure SQL DW.\nAzure Data Factory Data Flow: Building Slowly Changing Dimensions\nSCD Type 2 - why not?! But do build it with Data Flow!\nDataPlatformGeeks \u0026amp; SQLServerGeeks - YouTube channel\nAll webinars of our friends from India are recorded. You can find them all on youtube.\nThree reasons why Windows Server and SQL Server customers continue to choose Azure\nJulia White (T) is listing the most important benefits of choosing Azure.\nDetermine columns you don’t need using DMV’s in Power BI\nSimple trick presenting how to find them - with Kasper de Jonge (T).\nAnalysis Services Tabular Best Practices – Part 3\nConsider those tips collected out by Ginger Grant (T).\n","date":"2018-12-02T22:03:15Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/12/last-week-reading-2018-12-02/","title":"Last week reading (2018-12-02)"},{"content":"Good day SQL folks!\nHere is your weekly list of articles worth to read!\nFailover Detection Utility – Availability Group Failover Analysis Made Easy\nTo analyze the root cause for the failover of an Availability group, users are required to perform coordinated analysis of various logs including the Cluster Logs, SQL Error Logs, and the Availability groups extended events logs.\nConnecting Apache Kafka to Azure CosmosDB — Part II\nPart one gave a quick introduction into building a very simple, near real-time data integration pipeline between any JSON data that is written to Apache Kafka topics and Azure CosmosDB. The 2nd part of this blog post series will dig a little deeper in order to explore how to setup a data pipeline which allows to fully replicate all changes originating from various disparate data stores into CosmosDB.\nSSMS 18.0 Public Preview 5 Released\nThis is the announcement of the release of Public Preview 5 of SQL Server Management Studio (SSMS) 18.0. This release has a number of new features and capabilities and several bug fixes across SQL Server Management Objects (SMO), UI, etc.\nColumnstore Indexes – part 127 (“Batch Mode on Rowstore – is it a Columnstore Killer?”)\nGreat post as always. Check it out. Niko delivers great knowledge as always.\nSynchronizing Azure Cosmos DB Collections for Blazing Fast Queries\nAzure Cosmos DB is the fastest growing data service in Azure – and for good reason. The service offers global distribution in a few clicks, seamless horizontal scaling, automatic indexing, and 99.99% guarantees for availability, throughput, latency, and consistency. Enabling Cosmos DB multi-master mode provides a service-level agreement backed read and write availability of 99.999% - financially backed by Microsoft.\nCheers,\nDamian\n","date":"2018-11-25T21:34:53Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/11/last-week-reading-2018-11-25/","title":"Last week reading (2018-11-25)"},{"content":" Introduction Chris Webb is independent consultant specialising in Analysis Services, MDX, Power Pivot, DAX, Power Query and Power BI in general. He has been blogging about Microsoft BI for over 10 years, and racked up over 1000 posts on his blog! Also he speaks at a number of conferences including SQLBits, the PASS Summit, the PASS BA Conference, SQL Saturdays and user groups.\nThe author of \u0026ldquo;Power Query for Power BI and Excel\u0026rdquo;, a co-author of few other books and also he co-authored the white paper \u0026ldquo;Planning a Power BI Enterprise Deployment\u0026rdquo; for Microsoft.\nThis talk has taken place during Data Relay (formerly SQL Relay) conference in Birmingham (UK), on 10th October 2018 (Wednesday).\nInterviewers: Kamil Nowinski.\nHow did Chris Webb start working with SSAS? What is the future of SSAS On-Premise? How often the Direct Query has chance to being used in Power BI? What was the story of the beginnings of Chris\u0026rsquo; consultancy company?\nCheck these things and much more in this episode.\nAudio version Don\u0026rsquo;t you have time to read? You can listen this as a podcast! Wherever you are, whatever you use. Just use the player directly from this site (above), find it on Spreaker, Apple Podcasts, Spotify (new!) or simply download MP3. Enjoy!\nTranscript Kamil Nowinski: Hi Chris. We are sitting in this fantastic venue I think, during Data Relay right now.\nChris Webb: Yes, it’s difficult to say. I keep wanting to call it SQL Relay but it’s Data Relay.\nAbsolutely, yeah, so you just finished your session. Your 55-minute session about M language, yes?\nYes, the difficult thing about doing sessions that are 55 minutes is that I naturally think in one-hour sessions so there were about two slides at the end that I was able to very skillfully cut out at the last minute. But anything that isn’t an exact hour session if you’re a kind of experienced speaker is very, very off-putting. But you manage it anyway. It’s not a massive problem.\nScreen and ant. I mean\u0026hellip; speaker. Yeah, exactly. So how did you feel at that room, at that screen it was a very big screen. We are in the cinema, yeah?\nYeah, I know, to be honest I’m surprised that more events don’t take place at cinemas. I’ve done a few and having that big screen as an attendee is really amazing. I’m a heavy user of ZoomIt and actually I still want to use ZoomIt to point things out but actually you don’t even need ZoomIt because everything is so big. And for an attendee also the comfortable seats, the decent acoustics, it makes a massive difference. The one thing I would say, though, is a presenter having a big screen makes it actually a little bit more difficult because occasionally I want to see what is on the screen, not on my laptop screen.\nSo you extend the screen?\nNo, I duplicate, but if I’m standing out at the front, I might look back to see what’s on screen and actually you can’t see. And also it’s kind of nice to see whether the attendees can actually see the thing you’re pointing out. And from the big screen\u0026hellip; I couldn’t actually see anything that was on the on the big screen cause I was too close to it. But those are kind of minor things. From an attendee’s point of view having a gigantic screen is absolutely amazing. And you know, it worked so well. And they’re great venues as well.\nIt’s the best I’ve ever seen, so far at least. So I attached to this blog post the picture I took during your session, so the readers will be able to see how big the screen was.\nOK, let’s go back to the beginning. Could you tell me what your name is and where you live?\nSo, my name is Chris Webb, I live in the UK, just northwest of London, in a small town called Amersham. I’m a consultant and trainer. I’m kind of self-employed. I’m a one-man company and I travel all around the UK and Europe doing training consultancy, mostly nowadays on Power BI. In the past more Analysis Services. When I first started out, I was a purely Analysis Services Multidimensional person, and then Tabular and Power Pivot and Power BI came along so I moved with the times.\nYeah, I saved this question for later but when we’re talking about SSAS, I saw this tweet from you that you’ve started basically at the beginning of SSAS, when it appeared, it was almost 20 years.\n1998. Obviously I was very, very, very young in 1998. But yeah, it was kind of in my very first IT job, I was doing VB6 programming and then I got put on a project where we were evaluating OLAP services at the beta stage. So I was working for a company who’d actually already got lots of experience with various different OLAP tools, built their own OLAP tools for delivering data, and they were very sophisticated. They had lots of very complex requirements. So as a result, we spend a long time evaluating OLAP services, learning it, learning how to recreate lots of complex calculations which is how I learned MDX. And then once I’d done, I suddenly realised: “hey, now I know MDX and OLAP services,” which then became Analysis Services, which then became very, very popular. And that’s been my career pretty much ever since.\nSo you started learning MDX from the beginning basically. So you are an expert in that language.\nBack with the first betas. And it’s surprising how little things have changed. The fundamentals of the language are basically the same as they were 20 years ago. It’s not very fashionable anymore. Now everybody’s into Dax, M, and Python, and all of the other cool things. But even though I spend the majority of my time doing Power BI, I still reckon I spend about 20-30 percent of my time doing Analysis Services Multidimensional and MDX stuff.\nLet’s focus on that one and repeat, as you mentioned, you have your own consultancy company. What do you do for a living exactly? What do you do in this company? You mentioned that you’re trained a lot of workers.\nI’m a one-man band. I don’t do long contracts. The longest I’ll ever spend with a customer at a time is about 5 days. And probably on average my average engagement is about 2 days I would say. And literally, people contact me and say: “hey, Chris, I’ve got a problem with Power BI or Analysis Services” or “I’ve got a shopping list of problems, can you come in for a day or two and we can work through them and even tell me the answer?” So I don’t build stuff for people. I come in and kind of help them build stuff. And at the same time people will say: “oh, you know, we need some Power BI training or Analysis Services training for the team, can you come in and run a private course and cover these subjects?” So I do kind of a customised private training courses for people as well. And there’s a lot of overlap. Sometimes the consultancy I do turns into training because if you’re explaining how to solve a problem, you’ve got to explain the concept.\nSo you’re covering SSAS, Tabular, Multidimensional, Power BI, M language, did I miss something?\nWell, I suppose I do Excel and Power Pivot and Power Query and things like that. And now that Power Query is being included in Flow, maybe I’ll do a bit of Flow. But I wouldn’t paint myself as a Flow expert by any means. But yeah, anything where that basic Analysis Services engine and the Power Query engine is being used, I’ll go there and work.\nDoes it look promising? I mean the M language in, sorry, Power Query and M language in Microsoft Flow?\nYeah, I mean I think there’s gonna be all kinds of fun things you can do with it. I mean, at the moment it’s very, very limited. At the moment the Power Query integration is limited to SQL Server data sources, so you can connect to a SQL Server database, generate a name query, get some data out, and then do whatever you can do in Flow. So I wrote a blog post recently where it showed how you could run a query and then save it as a CSV file in OneDrive for Business. And actually a lot of people want to do that kind of thing. I’ve seen numerous times situations where people basically want to use Power Query in Excel or Power Query in Power BI Desktop to just do one of those kind of standard CSV data dumps, because they need a CSV file for some other system. And actually doing this in flow makes an awful lot more sense because you can save directly to the CSV file and you can set it to run on a schedule.\nIt’s completely easy, without any line of code.\nExactly, exactly.\nWhat do you think, what is the future of typical SSAS Multidimensional or Tabular model?\nYou ask more controversial questions, don’t you? OK, so in the near future, and I can say this because Amir Netz [T] said it on Twitter, Multidimensional will be going to the cloud. Now, the way things are at the moment we have Power BI and we have Azure Analysis Services. And based on the announcements that have been made at Ignite recently, Azure Analysis Services is not necessarily going away, but it looks like all of the interesting stuff is being put into Power BI Premium. From a technical point of view, there isn’t really that much difference between Azure Analysis Services and the engine behind Power BI Premium. So from Microsoft’s point of view, I can see why it’s a bit stupid really to have Power BI Premium and Azure Analysis Services, and why it makes a lot more sense to simplify the story and just say “we’ll just consolidate with Power BI Premium.” It’s also clear that more and more features are being added about Power BI Premium. It looks like Power BI Premium is basically going to be the kind of one-stop shop, where an enterprise can pay a set amount of money a month and get a full stack Microsoft BI implementation. Because we’re getting reporting services in the cloud as a feature of Power BI Premium, and I would guess that when Analysis Services Multidimensional comes to the cloud, it will come as a feature of Power BI Premium. That doesn’t necessarily mean all these features will always be limited to Power BI Premium. The way I understand it, for technical reasons it’s easier to put them in Premium first and then eventually these features will be coming to Pro in some shape or form. But it seems like if you’re an enterprise anyway, you would want to buy Premium for various other reasons as well. So I guess that’s where things are going. I mean we’ve had 20 years of Analysis Services and it’s still very, very strong and healthy as a platform. It’s just that nowadays it’s kind of hidden inside Power BI rather than its own branded thing so much. But you know the Analysis Services dev team still do an amazing work. Microsoft is still very, very serious about it. It’s just now people don’t even know it’s Analysis Services.\nYeah, true, exactly. They basically don’t know what is there behind the scenes. I’m observing that more and more companies are interested in Power BI or trying to adopt the Power BI in their enterprise. How does it look from your perspective?\nFrom my point of view, it’s absolutely crazy. I’ve never seen any Microsoft product that I’ve worked with become so popular as Power BI. Now, Power BI has obviously been around in lots of different shapes and forms for many years. It’s really since it got rebooted, what, 3 years ago in the shape and form that it is today, with Power BI Desktop and the separate Power BI service, that was the point where everything really kicked off. And, since then, the number of companies that I’ve seen who were just adopting Power BI, partly because of the price, because it’s cost-effective, partly because it’s native cloud which is also great, has obviously lots of advantages, and partly because it’s a really good BI platform.\nAnd developing so fast.\nExactly. I think for people who’ve worked in the Microsoft space, we’ve all been used to Microsoft in the past wasting its money on crazy stuff like the Zune or buying Nokia, which obviously did it no good whatsoever. And then the products that we were actually working with, people were buying, never seem to have quite enough love or money spent on them. I guess now, in the age of Satya Nadella [T], sanity has returned. Microsoft is spending its vast amounts of money on Power BI and, as a result, Power BI is just growing and growing and getting better and better at an amazing rate. And the end result is customers see this, they love what they’ve got and they just want to buy, buy, buy. I mean, in a way, there’s actually a real downside to this in that Power BI is moving so quickly. If you’re a somebody who works with Power BI like me, you just end up being completely overwhelmed. Every month, every week, there is something new, some substantial new feature to learn. And you just can’t keep up. You feel like you go away on a holiday for a month and suddenly the world is completely changed, and you’ve been left behind. It can be a little bit stressful trying to keep up with the massive pace of change.\nEvery month the Power BI team publish a new post about the last month’s changes, yes?\nYeah. I think it’s interesting to go back to 20 years ago, I can still remember around the year 2000, that was the point where there were still people who could claim to be complete SQL Server platform experts. They knew everything there was about every single aspect of SQL Server. But then after that, especially with SQL Server 2005, the world split into people who knew the Relational Engine, people who knew Analysis Services, people who knew Integration Services - you couldn’t possibly be an expert on everything. And I think actually Power BI is getting like that. It’s getting so big and so rich and mature that there will be\u0026hellip; I mean already, you see Marco and Alberto who are the DAX and Modelling experts. I would like to think that if somebody was looking for help with Power Query and M, they might come to me. There are going to be people who are experts on different bits of Power BI. Nobody is going to be great about the whole thing.\nSo you were even thinking about Power BI only, I was thinking about other areas like, you know, currently SQL engine it’s not only SQL engine. Like you mentioned previously, we still have Integration Services, Reporting Services. But now we have even more in Azure, like Azure Data Factory, Data Lake, Data Bricks and etc. There’s so much new things that you need to get and understand.\nExactly. And I think basic Power BI is a skill that anyone can learn. I mean, anybody with an IT background will be able to pick up Power BI very easily. That’s the point of it almost, it’s meant for non-IT users and that’s definitely true. But for people who want to be Power BI experts, there’s just more in Power BI than any one person I think can ever truly learn, or be good at.\nIn terms of Power BI and very popular DevOps approach, how can customers adopt DevOps for Power BI report deployment, and why they still can’t do that?\nI mean the story about managing Power BI report deployments is not good. The good thing is that Microsoft knows and understands that this is a problem and they are working on the solution. So, for example, again in the last month, what we’ve seen is\u0026hellip; there’s a very well-known tool called BISM Normalizer that Analysis Services Tabular developers have used for a long time. That has now been converted for use with Power BI. Mac software, who do lots of custom visuals of Power BI, I think have been recruited by Microsoft and have created a version of BISM Normalizer that works with Power BI, so we’ll be able to do, for example, partial deployments and compares and things like that. Also, something else that has been worked on quite a lot within the private Power BI team are all the PowerShell cmdlets for Power BI which will allow us to move different Power BI components like reports and datasets from one workspace to another, so the story is getting better and better. I wouldn’t say that it’s there yet, and I would also say that a lot of these tools are probably a bit too technical for the average Power BI user to ever use. But again, if you’re thinking about an enterprise Power BI deployment, there will always be some IT department involvement and they can set up the processes with these new tools, that will actually give you some control over versions, reports and things like that.\nOkay, cool. How often do you use Direct Query in Power BI projects for your customers?\nI do use it, however I would say that in a lot of cases, people start off using Direct Query, and then subsequently find out that actually they’re better off with import mode or using a live connection instead. The problem with Direct Query is that it promises more than it can really deliver, in that people hear “oh yeah, I don’t need to copy the data into Power BI, I can just leave it where it is and it will just generate queries and, okay, well, that sounds like the best possible thing we can do.”\nLook quite fine if I have only one or two steps in the editor.\nExactly. But then of course you find out that actually they’ve got these complicated reports with 20 different visualisations on a page, with slices and things, and then they click on a slicer - 20 different queries, more than 20 different queries get run. And oh look, we’ve got hundreds of reports and these are basically all kind of ad-hoc queries, how can you tune all of these different queries? For example, can you promise that Power BI is going to generate the most optimal SQL query in the first place? No. No tool would ever do that. It’s good, it’s useful in real-time scenarios, it’s useful when you’ve got very large amounts of data. It’s not the answer to all things for everybody that some people initially think. Composite models, a new feature where you can mix Direct Query mode and import - that is obviously going to make Direct Query a lot more useful, as is the new aggregations feature as well.\nExactly, so we are talking about the aggregation feature that Microsoft introduced. It’s a very powerful feature in Power BI. You can keep the aggregated data.\nWe can go back again to 1998 where Analysis Services Multidimensional had aggregations to speed things up. Finally, Power BI and Analysis Services Tabular has got the same things. I mean, to be fair, actually aggregations in Power BI is a bit more like HOLAP, rather than ROLAP in Analysis Services Multidimensional. And actually, if you ever used whole app in Multidimensional, you realise that there were some fundamental flaws with how it was implemented, which meant that it was totally, completely useless and the same flaws have been understood and avoided by Microsoft, so aggregations and composite models are a lot more useful. And the implementation is just fundamentally better than ROLAP and HOLAP was in Multidimentional.\nBut currently, you can have Direct Query to show the detailed version of the data set, yes?\nExactly. So I think it’s a very, very exciting development for people who are working with large amounts of data.\nWhen I first saw it, I just thought that this future revolutionised the capabilities in Power BI, so it’s quite a new feature. Okay, quite recently during the local group in London, you presented Q\u0026amp;A in Power BI. What’s new has appeared in that area?\nQ\u0026amp;A is one of those features that actually for years, for most of my time working with Power BI, I was a bit contentious of it. I just thought it was a bit of a demo feature and then, in the last couple of months some new features got added, which actually made me stop and think again that maybe Q\u0026amp;A was worth a closer look, or worth me looking at again. So the first thing that happened was\u0026hellip; one of the problems I’d always had with QA was not the feature, but where it was surfaced. There’s been a lot of confusion in Power BI about workspaces and who’s building what, where, and actually the ways that Q\u0026amp;A was surfaced and made available to the end-user meant that actually they weren’t really places that the end user would actually really want to use Q\u0026amp;A. Whereas in the last six months, we suddenly got to be able to drop a Q\u0026amp;A button on a report, so a user who is viewing a report and then thinks “I want to find out some more about that data,” would be able to click the button, type a question and then see a visualisation that answers their question, which for the first time I think is actually a sensible way of exposing Q\u0026amp;A. So it’s kind useful for end-users now in that respect. And then the second thing that happened was that we’ve got the ability to edit the linguistic schema. Now, this looks horrendously complicated and it is, but as soon as you look at it, you realise that it does really extend the possibilities of Q\u0026amp;A and means that a lot more end-user questions can be answered in a meaningful way. It’s quite a technical job to do, but then again as an IT consultant, I always like that the more technical, difficult things, because if they’re difficult, then my customers will probably need help with them and I can sell some training and consultancy to them to help them through it. That’s suddenly really extended the capabilities of Q\u0026amp;A. And so put those two things together, I suddenly thought “actually maybe I shouldn’t dismiss it, maybe it needs a closer look.” The more I looked at it, the more I was actually very, very impressed by it. So I hope Microsoft still keep investing in it. They’ve been investing in it for a long, long time now. It would be a shame if they lost interest now, it’s suddenly properly useful and I would say, for anybody that’s working with Power BI in enterprise environment, it probably is now worth your while spending maybe a day just tuning your Power BI data set for Q\u0026amp;A, maybe creating a separate data set for Q\u0026amp;A as well, which is specifically tuned, and then giving it to some of your end users and seeing whether they get any value from it. Because now I’ve got a suspicion that they might well do.\nI’ve seen this session as I mentioned, but I’m just wondering if you published that session or will you present that session somewhere or can we find it recorded?\nI don’t think it’s recorded anywhere. I’ve only done it twice now, so I do a lot of conferences and user groups and every six months or so, I sit down and think “oh, now everybody’s seen all my previous sessions, I’ve got to write a new session.” And it’s generally about the cool, new thing or something that I’m interested in and it’s just that this autumn I suddenly thought “well it’s time to do a session on Q\u0026amp;A.” So it will probably be coming to a user group somewhere near you. But I think it works quite nicely as a session, it’s an interesting subject, and there’s a surprising amount of technical depth there.\nThe second part of the set of questions, less technical. What hints would you give to young people, who wanted to start working on the IT market?\nIn IT in general? It’s an interesting question.\nOr maybe in Power BI market if you would like to be more specific.\nThat’s probably a better question for me to answer. It does come back down to those fundamental questions of do you get an academic education, or do you go out and get real-world experience, and ultimately, you probably need a little bit of both. I think really what you need is enthusiasm and to be interested in what you’re doing, passion. Because with IT, however much formal education you get, either from university, or from training on the job, or if you’re lucky enough to have a mentor or somebody to help you, the real way that you learn anything is just working on your own and playing around with things. And I think if you’ve got passion, if you’re interested, if you’ve found something that you’re interested in, you’ll just naturally just want to open up your PC, start playing with it, start doing some fun stuff and that’s the way you learn. That’s the way I learn anyway. Quite often I’ll find myself in an evening or a weekend thinking about something, or I find some particularly interesting data and I think “wow I’d really like to do a Power BI report on that.” And as soon as you do that, it not only helps you learn new skills, it helps you consolidate things that you already think you know, but maybe you don’t know very well. So it’s practice, practice, practice, that’s what you need to get going in IT. I would also say, we’re at Data Relay at the moment, you get a lot of benefit from coming to community events and talking to people. In a lot of IT jobs, even if you’re not the only person in the IT department, you might be the only person in the IT department who does what you do. Thinking about Power BI, there might only be one person in the company who knows about Power BI and you’re the reporting guy, that means that you can’t really learn a lot from your colleagues about what you do. As a result, just the ability to meet people who do the same job as you do in other companies, that’s going to be really invaluable from a learning point of view, I think.\nWhat do you think, what are the benefits from the community if you’re talking to the people and that kind of event like this?\nWell, I mean, obviously you get lots of free training from great speakers and I saw some really good sessions today about things that are kind of relevant to what I do, but which I don’t really know anything about. So I learned about Temporal Tables, learned about Azure Data Factory. These are things that I know exist, but having sat down and watched a really good speaker explain them me for an hour, I feel actually a lot more confidence that I understand what is going on. So the sessions themselves are very good, but also it’s just meeting people, networking, hanging around, those chats before and after sessions over lunch, that’s the really important thing. And also, let’s face it - anybody in IT - you’ll learn only so much in one job, you really learn by going from job to job every couple of years, getting new challenges, new experiences, meeting new colleagues. And the best way, I think, of getting a new job and progressing in your career, is to network, and events like Data Relay are a great place to network. It’s where you meet people in your area who do the same thing as you do, who might have a job going and that’s how you can move up on a salary grade, learn some new technology, learn a new business, see some new data and so on.\nI think the same. If you stay at the same company for too many years, basically your career will not develop, yeah?\nYeah, exactly. I think, I’ve heard people say that anybody with 20 years IT experience in one company on their CV ends up having actually had two years of IT experience and then eighteen years of doing the same old thing over and over again, not learning anything new.\nWhat do you think about MVP those days?\nThe Microsoft MVP programme? I’ve been an MVP now for 12 years and I am continually very, very grateful for it. It gives me lots of great benefits like free Azure credits, secret access to NDA materials, which helps me a lot. I think also from a Microsoft point of view, it’s a genius idea by Microsoft to create the MVP programme, because it means that it incentivises people like me to come out and speak at conferences. I mean, I would probably do this anyway if I wasn’t an MVP because it’s something I enjoy doing and to be cynical, it’s a good place to meet customers and network. But this is, let’s face it, a whole load of people giving up their time for free to do marketing and community work for Microsoft and if you think about how much the MVP programme costs Microsoft and how much good it does Microsoft, not just in selling Microsoft products, but making people more successful with what they do with those Microsoft products, which, to be honest, is what really sells the product is the last successful project that sells the Microsoft licenses for the next project. That is absolutely incredibly beneficial for Microsoft and Microsoft has understood this for a long time and it runs the programme very, very well. I’m very grateful for what it gives me, but I am full of admiration for how Microsoft uses it for everybody’s benefit. It’s not like anybody’s being exploited, everybody’s winning from this. I’m winning, Microsoft’s winning, the people here are winning. It’s a virtuous circle, so do more of it.\nAnd this year, I’m not sure if for the whole world, but at least in the UK, a lot of people have not been rewarded, yes? What do you think, why?\nI think, to be honest, there’s always kind of inflation in terms of the numbers of MVPs. There are always some people whose careers change, their personal circumstances change. They’ve done a lot of great service, but maybe haven’t done anything in the previous year. And the thing that Microsoft are always very careful to say about the MVP award is that it’s for community work in the previous year. However good a name you’ve got, you’ve always got to think how much community work am I doing, have I done in the previous year, so it’s inevitable that at some point there will be people that drop out and I think maybe Microsoft had been quite kind in the past, and this year there’s a very high bar that’s been set, there are people who do lots of great work. Maybe it’s time to thin out the ranks a little bit, give some new people who’ve done a lot of work an opportunity. I’ve not come across anybody who’s bitter about losing their MVP award, most of the ex-MVPs that I know have been very grateful for their time and understand why they weren’t re-awarded. It’s gonna happen to everybody, one year it’s going to happen to me, so I’ll go and just take each year as it comes and I’m just grateful for what I get.\n20 years, yeah?\n12 years as an MVP.\nGreat! How do you prepare yourself for a speech? You do a lot of conferences\u0026hellip;\nIt’s an interesting question. I think I do a lot of speaking and, as a result, the way\u0026hellip; because I’ve got a lot of experience speaking, it’s changed the way that I would prepare for a speech. So when I first started speaking, you would put together a slide deck and then I would practice, practice, practice over and over again. Nowadays, I never run through a session until I actually do it. I’ll put together a slide deck and then I just have a gut feeling of roughly how long it will take. Probably prepare more material than I’ll need, but then be able to drop things out depending on how the timing’s working, and then, I don’t know, personally I feel like any kind of speaking session should be fairly natural, which is why I don’t want to prepare too much. I feel like now I can speak to a set amount of time, so long as I have the structure of the presentation prepared and the demos prepared, and I spend a lot of time making sure my demos work. That can be a lot of prep time. And I will always spend at least the hour immediately before a session going over my demo, my deck, so I know how everything’s structured and how all my demos should work. After that, I feel like I can just fill in the blanks and speak. But I think it comes with experience. I think in a way also I found that being an experienced speaker sometimes makes you a bit lazy, and it makes you think you can get away without preparing as well as you should and as a result, sometimes you don’t do as good a job as a speaker as you know you can, because you’ve just not bothered to prepare as well. So in the last couple of years, well maybe a couple of years ago, I found I was doing a bit too much and then I decided to kick myself up the arse and make sure I did properly prepare everything. I don’t want to be the person, who can speak and make it, but then not really have any good content. I want good content as well.\nThat would be a very good question, next one. There are many conferences or events these days. How often do you speak?\nI would say at least once a month, possibly more than once. I always like to prioritise events in the UK and events during the week. I travel a lot for my work and so it’s very difficult to justify being away from home at weekends. So I don’t actually do that many SQL Saturdays, I prefer to do evening user groups. I do evening user groups in the UK, because then I can just take it\u0026hellip; I’m self-employed, so I can have a day off, do some admin in the morning, drive to the user group in the evening, do the session, that’s fine. If I’m travelling for work, then I will actively look out for local user groups, so I can speak. And if I’m away it’s a choice between speaking at a user report or watching movies on my phone in a hotel room on my own. So speaking at a user group is going to be a lot more fun than that.\nThat’s good news for a lot of leaders of local user groups.\nWell, you know, I’ll go out there and ask them.\nSo where do you publish your calendar?\nActually, I’ve literally only done that in the last month. I’ve added a page to my blog, because it suddenly occurred to me that I probably should list all of the events that I’m speaking at. So after this, three days of Data Relay in Birmingham, Reading and Bristol in the UK. The next thing I’ve got is I’m doing a remote presentation for the North Carolina user group. I think on the 23rd of October, then I’m going to do the PASS Summit in the US. I make exceptions, there are some things that are so important, I’ve got to travel for them. But the PASS Summit is obviously the kind of top of the tree for speaking in SQL Server and if I get accepted, then I’m going to want to go to that. And then, after that, I’m doing a remote session for the Icelandic user group, I think in November as well. And then I can’t remember after that. Probably SQLBits. I’ll be going to SQLBits next year. I’ve just submitted, so hopefully I’ll get to speak there.\nAbsolutely, you will. So how about your work-life balance, how does it work for you?\nWell, that’s the thing. I mean, being self-employed, there is a delusion of being in control of your own destiny when you’re self-employed. What actually happens is that when you’re self-employed, you realise you’re responsible for bringing the money in and there’s nobody else that you can rely on. You’re more likely to accept work when somebody offers it to you, which then means you’re just more likely to kind of overwork yourself. I mean the other thing that I do is I travel an awful lot and you have to offset that as well. So that kind of goes back to what I was saying, trying to avoid doing anything at the weekends. Trying to make weekends family time. And also because I’m self-employed and only do these short-term engagements, actually I don’t get any work very much in August, because everybody’s away on holiday. So I just make sure I take pretty much the whole of August off to go on vacation, to spend time with the family. And similarly there are other times of the year, like the couple of weeks over Christmas, nobody ever does any work, so that’s holiday for me. I mean, I suppose if I was to list a number of actual days holiday I have, it’s quite generous, but then you count in all of the evenings I spend working, all of the Sunday evenings that I’ll be travelling somewhere, so I can be on-site to work on a Monday morning. I have a lot of holiday, but I do a lot of work as well.\nWhat does it look like during the week? Do you have some schedule like I’m working from 8:00 a.m. to 4:00 p.m.?\nIt’s nothing like that, it’s completely all over the place. For this week for example, on Monday I was on site with a customer, doing some training for them. It’s actually one out of five days of training but the five days are all split up, so that was about an hour and a half away from where I live, so that was an hour and a half’s drive, full day of training, hour and a half’s drive home. Then, in the evening on a day like that, I have to spend at least an hour doing emails, catching up with things, answering people who’ve asked me about more work, so you know, got to catch up with those things. Then on Tuesday, I had a half-hour call with a prospective customer at 9:30, and then at 10 o’clock, I was on the phone with another customer for an hour dealing with a Multidimensional MDX problem. Then I went to take my wife to our hospital appointments in the middle of the day. And then, when I came back, I had about 45 minutes on the phone with another customer. And then in the evening it was like “oh I’ve got Data Relay the next day, I’d better look at my slides and practice my demos” and things like that, so that was me in the evening. It’s completely non-structured. I don’t say that I’ve got kind of working hours and non-working hours. It all just blurs, unfortunately. Which is not a good thing to say.\nYeah, but it is what it is, yeah?\nYeah, it is, and I’m not gonna ever complain about the job that I’ve got. I get a lot of very interesting work, I really enjoy what I do and I get paid for it as well, so I would never complain about what I do. I’m very, very lucky.\nWould you recommend that type of work?\nYeah, I mean I would say that one of the big downsides is that being a single person company, you don’t have colleagues, you don’t have somewhere to go during the day. There is just you on your own and sometimes it gets a bit lonely, you know, there’s no Christmas party, there’s no hanging around chatting with people at the water cooler. There are customers I see reasonably frequently and then I’m friendly with them, but it’s not anything like a relationship you would have with a colleague at work, where you’re sitting next to somebody all day, every day and you can just have a chat with them. When somebody’s a customer, you’ve got a different relationship, you’ve got to be a little bit more professional, you can’t get out your phone and start playing a game or something in the middle of the day. You’ve got to take lunch when they want to take lunch. So yeah, it does get a bit lonely. Like I said also, those kind of lonely hotel rooms on a Wednesday night in Copenhagen, when you’re there on your own. There’s nobody to have dinner with, there’s nothing to do but emails. I always make sure I call my family at 7 o’clock every evening when I’m away, so I can at least have a chat with them, but it is quite a hard lifestyle, all the travel and all of the being on your own and always being customer-facing.\nI heard that sometimes it helps if you go to work to the public places. Did you practice that kind of thing sometimes?\nNo, because I like a bit of comfort and if I’m in a public place, I think what it does, working in a public place, is it puts you on your best behaviour. It means you concentrate more on work and do less browsing. But actually I think I’m fairly well disciplined anyway. I don’t have a problem doing the work in the evenings. I always, like I said, end up spending an hour or so, at least an hour or so doing stuff. But it’s just, you know, you’re doing it on your own and I’d rather be at home with my family rather in a hotel room on my own.\nHow long have you been working on the market as a consultant? When did you start with your company?\nThat was 12 years ago when I set up. I didn’t really think too much about it. When I think back to 12 years ago, I probably think “wow, God”. I didn’t even realise what I was doing, but I was in my last perm job, it was the usual story where a lot of stuff had been offshored and I’d still got my job, but I wasn’t doing\u0026hellip; you know, in the future I wasn’t going to be doing any hands-on work. I was just gonna be writing specs. And I like the hands-on technical stuff, so I thought “it’s time to go” and I thought “well, I’ll just set up a company”. I’d already been blogging for a couple of years, so I thought “well I’ll just put something on my blog and say that I’m available to do consultancy work and if the work comes, then great. If it doesn’t - well, I’ll just go and get a contract and work for six months a year like contractors normally do.” And then, in the first month, I had a couple of days of work, it worked out pretty well, and then it just built on from there. So I was just very, very lucky and I’m glad that it worked out for me. I think it helped that I already got my blog. That allowed the kind of publicise myself and get the work that way.\nYes, so you basically didn’t plan very well. You had a blog and for a couple years before, but generally you didn’t plan it.\nExactly. I think nowadays you go to, well, I think at the PASS Summit this year there will be a session on building your personal brand, and there are some people, for example Brent Ozar who does a great job, a very, very professional job with his personal brand and everything. And my marketing efforts are nowhere near as good as that. But at the end of the day, the personal brand doesn’t make you any money. It’s the work you actually do that makes you money, and the personal brand, any marketing activities you do, ultimately, you’ve got to do enough to drive work to keep you busy. If you spend your whole time making your blog look pretty or getting some great avatars and things together, or doing podcasts, then you don’t have enough time to actually make the money.\nMaybe this is just another way how to achieve the goal. But I don’t agree that you don’t have the brand. You have your blog and I think your blog is very nice.\nIt’s not very slick and professional, though. When I look at some people, especially the new generation of people, who are doing stuff on YouTube. I mean, I know Adam Saxton, Guy in a Cube (T), is a Microsoft employee, but his branding, his slickness is so, so good. And there are lots of other\u0026hellip;\nHe publishes three times a week if I remember well\u0026hellip;\nExactly. But there are lots of other people out there who blog, who do YouTube videos, who do a much slicker job than I do with their personal branding. But again, I kind of see it as a means to it. Maybe I’m just lazy. I’m not very good at making things look, making YouTube videos or making nice-looking websites, or something like that.\nI think you are just focusing on different things.\nI know I just do the things I do. It’s worked in the past and hopefully will work in the future. If I find something interesting, when I’m doing my kind of day-to-day job, I’ll blog about it. It serves as marketing, it also serves as a way of helping me remember the things that I’ve learned, because if I didn’t write it down in a nice structured, central, easily searchable way, I’d forget about it and then the next time somebody asked me the same question, I’d think, “oh, I used to know that” but then I’d have forgotten. I am the number one person that hits my blog. I google for myself several times every day just to look up the stuff that I know I’ve written down, but I’ve forgotten otherwise. All the traffic is from me. And a lot of the time I’m googling for something and I’ll find one of my blog posts, and I forgot I’d even written about it.\nWhich is very good! So, what is the recipe to achieve the success if you have your own company?\nI wish I knew. I think the only recipe that I’ve got is I do what I do, it works for me and it’s just worked out. So much of it depends on you, your personality, what you enjoy doing, your own particular strengths and weaknesses. A recipe that works for one person probably wouldn’t work for anybody else. But I don’t know, I mean there’s some standard good business things, which means that if somebody emails me asking about some work, I’m going to hopefully reply to them within 24 hours and get back to them. If somebody wants me to turn up to do some training, I make damn sure that I’m going to turn up and be there on time to do the training and do a professional job. The best way to build a business is to get satisfied customers who then come back to you and give you more business in the future. And it’s the same whether you’re running an IT consultancy or training business or whether you’re running a restaurant, or whether you’re running anything else. Getting the customers is the hard part. The way you build a business is get new customers but also keep the existing customers coming back. You do a bad job, you’re going to be in trouble.\nSo we are perfectly flying to the next question, the last one. Are you a perfectionist?\nNo. Actually, maybe in some things I’m a perfectionist, but I would say, in general, no. I am the opposite of a perfectionist. I like to find a solution to a problem, but in my personal life, if you saw my office at home, it is incredibly messy. Some people have this perfectly clean, well-organized personal office with four screens, all arranged for maximum productivity. That’s not me. I’ve got a knackered old office chair, I work off my laptop half of the time, ‘cause it’s just easier.\nYou don’t have any external monitor?\nI do, but I don’t use it most of the time. I just work off my laptop. I’m surrounded by mess. I know where everything is, you know, I’m one of these kinds of messy, creative type people. But I’m not going to spend time optimizing absolutely everything. Give me an MDX calculation though, you give me a bit of M, I do like to make sure my code is nicely indented. There are a couple of things, especially about MDX, which I’ve been doing for years, where I like to have my MDX written in a particular way, because I know that just works. And it’s not just a kind of OCD thing. I’ve seen lots of cases where people will write something that seems okay, but it’s not the way I would have written it. And it just doesn’t work properly or it doesn’t perform well. And then, when I write it the way I would have written it, it works well, you then get to the bottom of it and find that actually there was something weird going on in the engine. But I just have a set way of doing things that I like, because I know it’s worked well in the past. And I’ll make sure I do it the way I like it. So I’m not necessarily perfectionism, but I like to do a good job, clean code and deliver something that’s good quality.\nAnd we can see it during your presentations, absolutely. So at the end of our conversation, tell us where we can find you.\nSo, on Twitter I am @cwebb_bi and my blog is blog.crossjoin.co.uk and my website, if you want to hire me for consultancy and training is www.crossjoin.co.uk.\nOkay, thank you very much, Chris, for this conversation.\nThank you very much, it’s been a pleasure.\nUseful links Chris\u0026rsquo; twitter: @CWebb_BI\nChris\u0026rsquo; websites: BLOG | Company Site\nTools: BISM Normalizer | ZoomIt\n","date":"2018-11-23T00:03:34Z","image":"/uploads/2018/11/ASF016_DataRelay_with_Chris_Webb.jpg","permalink":"/2018/11/asf-016-chris-webb-interview/","title":"ASF 016: Chris Webb interview"},{"content":"Today I would like to show how to implement temporal table solution when you have an existing table already present in the database. But this will not be all you find in the article. At the end I would like to show you an example where a new column is added to temporal table.\nThe previous parts can be found here:\nPart 1\nPart 2\nI will not talk about a scenario where a new empty table is created first and then data from existing table are transferred. That is not the preferred way of dealing with temporal tables and in fact that is even worse scenario I can think about.\nLet’s do it in the right way. The table I created has the following structure:\nCREATE TABLE dbo.ProductsV3 ( ProductId INT IDENTITY NOT NULL PRIMARY KEY, Name NVARCHAR(100) NOT NULL, Number INT NOT NULL, Price MONEY NOT NULL ) As you can see this is a regular table without the columns that are necessary to make it a temporal table. First of all let’s load some data to this table. That will emulate the real environment we usually have:\nINSERT INTO [dbo].[ProductsV3]([Name],[Number],[Price]) SELECT [Name],[Number],[Price] FROM [dbo].[ProductsV2] GO By the way – the table ProductV2 is a temporal table we have used in last blog post. According to the definition the temporal table contains two tables: the table that holds the current data and the historical table with changed records. The statement above will transfer only the actual data without touching the historical table. That makes sense of course but I was asked about that this week. Here is the query execution plan to make it clear:\nAccording to the Books Online the process of converting a regular table to the temporal one should be done in transaction and should have the following steps:\n• Add two non nullable columns of type datetime2 to the existing table. The precision used in the columns is up to you.\no The columns must have default constraint turned on – as we already have data in the table\n• Mark the two columns as PERIOD FOR SYSTEM_TIME\n• Drop the defaults as SQL Server will take care about managing data in these column.\n• Set SYSTEM_VERSIONING to ON and add historical table name or not if you wish the SQL Server does it for you\nHere is the code:\nBEGIN TRAN; ALTER TABLE dbo.ProductsV3 ADD SYSSTART DATETIME2(0) GENERATED ALWAYS AS ROW START NOT NULL CONSTRAINT DT_ProductsV3_sysstart DEFAULT(\u0026#39;19000101\u0026#39;), SYSEND DATETIME2(0) GENERATED ALWAYS AS ROW END NOT NULL CONSTRAINT DT_ProductsV3_sysend DEFAULT(\u0026#39;99991231 23:59:59\u0026#39;), PERIOD FOR SYSTEM_TIME (SYSSTART, SYSEND); ALTER TABLE dbo.ProductsV3 DROP CONSTRAINT DT_ProductsV3_sysstart ALTER TABLE dbo.ProductsV3 DROP CONSTRAINT DT_ProductsV3_sysend ALTER TABLE dbo.ProductsV3 SET ( SYSTEM_VERSIONING = ON ( HISTORY_TABLE = dbo.ProductsV3Hist ) ); COMMIT TRAN; And that is it! No sweat at all! Now let’s go to the other situation where the table is already a temporal one and we have a new column to be added to the table. I bet you already can imagine that code will not work as long as the ProductsV3 table is a temporal table:\nALTER TABLE dbo.ProductsV3 ADD Colour NVARCHAR(20) NULL The error message says the same:\nMsg 13550, Level 16, State 1, Line 156 Add column operation failed on table \u0026#39;TemporalDemo.dbo.ProductsV3\u0026#39; because it is not supported operation on system-versioned temporal tables. Let’s try another way then. Let’s remove the system versioning from the ProductsV3 table, add the column and turn on the system versioning again. I will do that operation in transaction so anything wrong happen in between the whole operation is not accepted:\nBEGIN TRAN; ALTER TABLE dbo.ProductsV3 SET ( SYSTEM_VERSIONING = OFF) ALTER TABLE dbo.ProductsV3 ADD Colour NVARCHAR(20) NULL ALTER TABLE dbo.ProductsV3 SET ( SYSTEM_VERSIONING = ON ( HISTORY_TABLE = dbo.ProductsV3Hist , DATA_CONSISTENCY_CHECK = ON ) ); COMMIT TRAN; The error message says:\nMsg 13523, Level 16, State 1, Line 168 Setting SYSTEM_VERSIONING to ON failed because table \u0026#39;TemporalDemo.dbo.ProductsV3\u0026#39; has 7 columns and table \u0026#39;TemporalDemo.dbo.ProductsV3Hist\u0026#39; has 6 columns. That leads me to the idea that the new column must be added to both tables in the same transaction. That means that both tables must have exactly the same structure. I hope you can see that I also included a phrase: DATA_CONSISTENCY_CHECK = ON along with the HISTORY_TABLE. This data check just want to find out if the column that is defined as a SysEndTime has greater or at least equal values to the column that is defined as a SysStartTime:\nSysEndTime ≥SysStartTime\nBy the way when the SYSTEM_VERSIONING is going to be turned on the SQL Server checks the following (according the official documentation):\n• The number of columns is the same in both the current and the history table.\n• The datatypes match for each column between the current and the history table.\n• The period columns are set to NOT NULL.\n• The current table has a primary key constraint and the history table does not have a primary key constraint.\n• No IDENTITY columns are defined in the history table.\n• No triggers are defined in the history table.\n• If the current table has period columns specified as HIDDEN, the matching columns in the history table are also hidden.\n• No foreign keys are defined in the history table.\n• No table or column constraints are defined on the history table. However, default column values on the history table are permitted.\n• History table is not placed in a read-only filegroup.\n• The history table is not configured for change tracking and change data capture.\nYou might guess that this code will not work, too. I just tried to add column to both tables but the data size in the columns are not equal:\nBEGIN TRAN; ALTER TABLE dbo.ProductsV3 SET ( SYSTEM_VERSIONING = OFF) ALTER TABLE dbo.ProductsV3 ADD Colour NVARCHAR(20) NULL ALTER TABLE dbo.ProductsV3Hist ADD Colour NVARCHAR(10) NULL ALTER TABLE dbo.ProductsV3 SET ( SYSTEM_VERSIONING = ON ( HISTORY_TABLE = dbo.ProductsV3Hist, DATA_CONSISTENCY_CHECK = ON )); COMMIT TRAN; The error message says:\nMsg 13525, Level 16, State 1, Line 202 Setting SYSTEM_VERSIONING to ON failed because column \u0026#39;Colour\u0026#39; has data type nvarchar(10) in history table \u0026#39;TemporalDemo.dbo.ProductsV3Hist\u0026#39; which is different from corresponding column type nvarchar(20) in table \u0026#39;TemporalDemo.dbo.ProductsV3\u0026#39;. What I also tried but without success is:\nBEGIN TRAN; ALTER TABLE dbo.ProductsV3 SET ( SYSTEM_VERSIONING = OFF) ALTER TABLE dbo.ProductsV3 ADD Colour NVARCHAR(20) NULL ALTER TABLE dbo.ProductsV3Hist ADD Colour NVARCHAR(50) NULL ALTER TABLE dbo.ProductsV3 SET ( SYSTEM_VERSIONING = ON ( HISTORY_TABLE = dbo.ProductsV3Hist, DATA_CONSISTENCY_CHECK = ON )); COMMIT TRAN; The error message is very similar to the one displayed last time:\nMsg 13525, Level 16, State 1, Line 220 Setting SYSTEM_VERSIONING to ON failed because column \u0026#39;Colour\u0026#39; has data type nvarchar(50) in history table \u0026#39;TemporalDemo.dbo.ProductsV3Hist\u0026#39; which is different from corresponding column type nvarchar(20) in table \u0026#39;TemporalDemo.dbo.ProductsV3\u0026#39;. At the end we have to add the column that way to make sure in both tables this column is identical:\nBEGIN TRAN; ALTER TABLE dbo.ProductsV3 SET ( SYSTEM_VERSIONING = OFF) ALTER TABLE dbo.ProductsV3 ADD Colour NVARCHAR(20) NULL ALTER TABLE dbo.ProductsV3Hist ADD Colour NVARCHAR(20) NULL ALTER TABLE dbo.ProductsV3 SET ( SYSTEM_VERSIONING = ON ( HISTORY_TABLE = dbo.ProductsV3Hist, DATA_CONSISTENCY_CHECK = ON )); COMMIT TRAN; What to do in case of removing a column from the temporal table? Well you have to follow the same pattern:\nBEGIN TRAN; ALTER TABLE dbo.ProductsV3 SET ( SYSTEM_VERSIONING = OFF) ALTER TABLE dbo.ProductsV3 DROP COLUMN Colour ALTER TABLE dbo.ProductsV3Hist DROP COLUMN Colour ALTER TABLE dbo.ProductsV3 SET ( SYSTEM_VERSIONING = ON ( HISTORY_TABLE = dbo.ProductsV3Hist, DATA_CONSISTENCY_CHECK = ON )); COMMIT TRAN; And the same pattern is also valid when you would like to change the data type of a column.\nThat is all for today! Stay tuned – in the next blog post I will try to show what happens when you do a SELECT statement on the temporal table.\nCheers\nDamian\n","date":"2018-11-20T08:00:24Z","image":"/uploads/2018/11/P1988170.jpg","permalink":"/2018/11/starting-from-zero-temporal-tables-part-3/","title":"Starting from zero – Temporal tables, part 3"},{"content":"No, you don’t need a blockchain\nThomas LaRock (T) says: \u0026ldquo;Don’t buy a blockchain solution unless you know for certain you need one.\u0026rdquo;\nAnatomy of a spool\nHugo (T) collected all (undocumented) details about all kind of spools in one place.\nCreating a data hub for your analytics with PolyBase\nMore connectors for Polybase in SQL Server 2019 CTP 2.0.\nFind out how to share a self-hosted Integration Runtime (IR) across multiple data factories.\nPaul White Blog\nTemporary site where Paul (T) put down his archival posts from SQLBlog.com\nInteracting with SQL Server 2019 Big Data Clusters\nPaul\u0026rsquo;s (Andrew, this time - T) quick overview.\nHow to handle SSRS multi-value parameter filtering in SQL Server Parallel Data Warehouse\nThree limitations in Azure SQL DW which causing the issue.\n20 Years Of Analysis Services\nThat\u0026rsquo;s right - it has happened on Friday, 16th of November. Can I say that Chris Webb (T) is a father?\n","date":"2018-11-18T21:56:52Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/11/last-week-reading-2018-11-18/","title":"Last week reading (2018-11-18)"},{"content":"This article is the second one in a series. You can find part one on our blog. This time I would like to start with defining one more table that will be used later in the demos. The dbo.ProductV3 table will be a little bit different than the previous two. I will use this table for a different scenario. The previously created tables will be useful when we start with temporal functionality for new tables. But what in case when we need to transform the existing table into temporal table? That is the scenario I would like to test on the table below:\n1 2 3 4 5 6 7 8 CREATE TABLE dbo.ProductsV3 ( ProductId INT IDENTITY NOT NULL PRIMARY KEY, Name NVARCHAR(100) NOT NULL, Number INT NOT NULL, Price MONEY NOT NULL ) GO Today we will check how to load data into the temporal table (insert statement) and I will point ou t to one very important thing during that process.\nLoading data to the temporal table should be pretty easy. If you think about this process a little you know that it should only affect the base table and have zero effect on the historical table. I load one row to the to the ProductsV2 table\n1 2 3 INSERT INTO [dbo].[ProductsV2]([Name],[Number],[Price]) VALUES(N\u0026#39;ProductA\u0026#39;,123000,5.87) GO The query plan is exactly as we might expect. There is a clustered index insert on the base table and the historical table was not touched during the INSERT operation:\nIf you do a SELECT statement on the ProductV2 table than you will see that one record was inserted (no surprise on that). Please take a look on the two system date columns. The SYSSTART column contains the datetime of the operation and the SYSEND column is set to 9999-12-31 23:59:59.9 in my case. Please bear in mind that the precision of the data type in the column was set during the table creation.\n1 2 SELECT * FROM dbo.[ProductsV2] GO If you do the same select statement against the ProductsV2Hist table than you will see that nothing is in that table so far.\nNow let’s load multiple rows into the ProductsV2 table. The question is what will be the value of datetime stored in the SYSSTART column:\n1 2 3 4 5 6 7 INSERT INTO [dbo].[ProductsV2]([Name],[Number],[Price]) VALUES(N\u0026#39;ProductB\u0026#39;,223000,1.00), (N\u0026#39;ProductC\u0026#39;,323000,2.00), (N\u0026#39;ProductD\u0026#39;,423000,3.00), (N\u0026#39;ProductE\u0026#39;,523000,4.00), (N\u0026#39;ProductF\u0026#39;,623000,5.00) GO The thing I would like to check now is how are the SYSSTART column values handled in the transaction during the data loading process. Let’s take a look at the code:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 BEGIN TRANSACTION PRINT CONCAT(\u0026#39;Begin of the transaction: \u0026#39;,SYSDATETIME()) WAITFOR DELAY \u0026#39;00:00:05\u0026#39; PRINT CONCAT(\u0026#39;After the first waiting: \u0026#39;,SYSDATETIME()) INSERT INTO [dbo].[ProductsV2]([Name],[Number],[Price]) VALUES(N\u0026#39;ProductG\u0026#39;,723000,10.00) PRINT CONCAT(\u0026#39;The ProductG inserted at: \u0026#39;,SYSDATETIME()) WAITFOR DELAY \u0026#39;00:00:05\u0026#39; PRINT CONCAT(\u0026#39;After the second waiting: \u0026#39;,SYSDATETIME()) INSERT INTO [dbo].[ProductsV2]([Name],[Number],[Price]) VALUES(N\u0026#39;ProductH\u0026#39;,823000,11.00) PRINT CONCAT(\u0026#39;The ProductH inserted at: \u0026#39;,SYSDATETIME()) WAITFOR DELAY \u0026#39;00:00:05\u0026#39; PRINT CONCAT(\u0026#39;Commiting transaction: \u0026#39;,SYSDATETIME()) COMMIT GO In the first line I start a transaction. Then there is a WAITFOR statement which emulates business logic that for example could be done by a stored procedure. That a record for ProductG is inserted to the table and again we have to wait for 5 second and so on. At the end I just commit the transaction. The question is – what will be the value in the SYSSTART column for each insert operation?\nBelow you can find the result of the batch execution:\n1 2 3 4 5 6 7 8 9 10 11 Begin of the transaction: 2018-11-17 08:40:25.5248095 After the first waiting: 2018-11-17 08:40:30.5411411 The ProductG inserted at: 2018-11-17 08:40:30.5411411 After the second waiting: 2018-11-17 08:40:35.5533124 The ProductH inserted at: 2018-11-17 08:40:35.6314396 Commiting transaction: 2018-11-17 08:40:40.6471049 What can we find in the table ProductsV2?\nIs that a surprise that both records has the SYSSTART value as it were at the beginning of the transaction? From the other hand we can see that the record were really inserted slightly later. But I am not surprised as the SYSSTART column is the system column so the value entered there is the system datetime.\nThe last experiment for today is to prove that it is not possible to use the system columns (SYSSTART and SYSEND) while load the data to the table. It makes sense to me as those column are handled by the SQL Server thus cannot be used (modified) directly. In this example I took data from ProductsV2 table and load the into the Products table:\n1 2 3 4 INSERT INTO [dbo].[Products]([Name],[Number],[Price],[SYSSTART],[SYSEND]) SELECT [Name],[Number],[Price],[SYSSTART],[SYSEND] FROM [dbo].[ProductsV2] GO The result is as follow:\n1 2 3 Msg 13536, Level 16, State 1, Line 100 Cannot insert an explicit value into a system period column in table \u0026#39;TemporalDemo.dbo.Products\u0026#39;. Use INSERT with a column list to exclude the system period column, or insert a DEFAULT into the system period column. We could load the data in this way:\n1 2 3 4 INSERT INTO [dbo].[Products]([Name],[Number],[Price]) SELECT [Name],[Number],[Price] FROM [dbo].[ProductsV2] GO The table Products contains 8 rows and the SYSSTART column values are the same but different from the SYSSTART values in the ProductsV2 table. Which is not surprised at all.\nThat is all for today. The next blog post will be about converting the existing table into a temporal one. I will also explain what could happen in between and how SQL Server checks if we are going to fool him during the conversion process. Stay tuned.\nCheers,\nDamian\n","date":"2018-11-16T13:23:42Z","image":"/uploads/2018/05/P1988106.jpg","permalink":"/2018/11/starting-from-zero-temporal-tables-part-2/","title":"Starting from zero - Temporal tables, part 2"},{"content":"\u0026ldquo;Power BI is just growing and growing and getting better and better at an amazing rate.\u0026rdquo;\nWho said that?\nWatch the video trailer and guess who is the next guest.\n","date":"2018-11-16T08:41:39Z","image":"/uploads/2018/11/ASF-016-video-invitation.jpg","permalink":"/2018/11/asf-016-video-invitation/","title":"ASF 016 video invitation"},{"content":"It\u0026rsquo;s T-SQL Tuesday #108. This month\u0026rsquo;s topic is hosted by Malathi Mahadevan (B|T) and she asks us to\u0026hellip;\n\u0026ldquo;pick one thing you want to learn that is not SQL Server. Write down ways and means to learn it and add it as another skill to your resume. If you are already learning it or know it – explain how you got there and how it has helped you. Your experience may help many others looking for guidance on this.\u0026rdquo;\nPASS Summit in Seattle took place last week and we know that SQL Server, not only new 2019 version, covers more areas then only SQL Engine. I love SQL Server and I love playing with data. But quite recently I adore playing with massive data. Big data. That\u0026rsquo;s why I\u0026rsquo;m more focusing on the Microsoft tools described below. Also, I\u0026rsquo;m really happy that gradually I realize my plan of learning established last year. Although, I haven\u0026rsquo;t used R language, Data Science and Machine Learning as I wished - I do it with other points on my list. And to tell the truth, I think that the original plan was too wide.\nPower BI I\u0026rsquo;m big fan of Power BI and I follow most news that appear in this area. Unfortunately, my role as a Data Engineer is not too much related to the tool. But whenever I can use Power BI to create quick and interactive report(s) - I\u0026rsquo;m using that opportunity.\nAzure SQL DataWarehouse Hmm\u0026hellip; It still has got \u0026ldquo;SQL\u0026rdquo; in name, but how much is SQL in that SQL? Hard to say as Azure SQL DW is not an ordinary SQL. It\u0026rsquo;s Massively Parallel Processing (MPP) architecture and that means you need to change your way of thinking and designing solutions in it. At the beginning of year 2018 I started working with SQL DW in Azure, doing some experimentation and, by the way, I prepared entire session about the topic. Quite recently I joined to private preview SSDT program where I\u0026rsquo;m able to test this tool for Azure SQL DW.\nHow to start playing with Azure SQL DW? Do begin reading documentation and then try to do some practice exercise. Be careful. This is expensive tool and fee is charged per hour upfront. Drop your database once you finish experimenting.\nAzure Data Factory v2 \u0026amp; Databricks Also, at the beginning of the year I started working with ADFv2. Version 2 is a way way better than v1. And I\u0026rsquo;m talking not only about UX. You can build very efficient ETL/ELT processes using a lot of available connections for source and sinks (destination). One of the components can run Databricks\u0026rsquo; notebook or JAR which giving you possibilities to extend the process and build business logic using Scala or Python.\nBut this is not the end of the story. New feature is quickly coming: Data Flow. If you remember Data Flow from SSIS - this is a counterpart. You can design all required transformation, calculation \u0026amp; aggregation without coding. And everything will be run by using Azure Databricks under the hood. I will be posting about this feature on my blog soon.\nIf you want to know more - read this documentation and ask ADF team for access to Data Flow in preview: adfdataflowext@microsoft.com\nHow to learn? These days, there are tons of sources and materials. The only problem remains is\u0026hellip; time.\nLet me mention you some of the ways how to learn:\n- Read Microsoft documentation: Power BI, Azure SQL DW, Azure Data Factory v2 Data Flow, Azure Databricks.\n- Find local Azure/SQL/Power BI local user group - they always happy to welcome new members and share knowledge\n- Learn from courses, e.g. edX.org\n- Go on conference. Don\u0026rsquo;t you have budget? Join to free one like SQL Saturday.\n- Stay updated. You can start with our Last Week Reading series.\nThis was my point of view and what I was learning during past 10 months. What is yours? Let me know - leave a comment.\nThanks for reading!\n","date":"2018-11-13T23:33:14Z","image":"/uploads/2018/11/austin-chan-275638-unsplash_1280.jpg","permalink":"/2018/11/t-sql-tuesday-108-non-sql-server-technologies/","title":"T-SQL Tuesday #108 – Non SQL Server Technologies"},{"content":"Hello folks, how are you today? It was a very comprehensive week and many of my friends returned home after PASS Summit. As always, PASS Summit is the event where you can hear many announcements. This post, predominantly, base on them. Unfortunately, I did attend the event this year, but MVP Summit (in March next year) will make it up to me.\nPASStv is your virtual ticket to PASS Summit 2018\nThis is the must see. Three days of selected videos.\nWhat are paginated reports in Power BI Premium? (Preview)\nPaginated reports, long the standard report format in SQL Server Reporting Services, are now available in the Power BI service.\nIntroducing: Power BI data prep with dataflows\nSo, data flow allows you to have ETL operations in a centralized manner. Self-service Data Warehouse in Power BI? Sounds good!\nAnnouncing: Preview of Machine Learning Services with R support in Azure SQL Database\nDo run R scripts and train machine learning models in Azure SQL Database by embedding them in T-SQL stored procedures.\nAzure SQL Data Warehouse introduces new productivity and security capabilities\nAll the exciting enterprise features announced for #AzureSQLDW - Row Level Security, Workload Importance, and Continuous Integration/Deployment amongst others!\nAzure SQL Data Warehouse provides frictionless development using SQL Server Data Tools\nIf you haven\u0026rsquo;t heard about it yet - this is great news and you can take an access to preview.\nWhat’s New in SQL Server 2019 CTP 2.1: Faster Functions\nA small piece of code where Brent [T] shows how a scalar function works under 2019.\nIntroducing Scalar UDF Inlining\nFurther expanding query processing capabilities with several new features under the Intelligent Query Processing (QP) feature family.\nRedgate State of Database DevOps Survey 2019\nTake part in the Redgate\u0026rsquo;s State of Database DevOps Survey. It\u0026rsquo;s the leading research on that area. Help here and do secure your chance to win a $250 Amazon voucher.\nSQLBits 2019 is coming!\nRegistration for SQLBits 2019 has been opened on Thursday (8/11). Will you be an Early Bird?\nHow do you deploy machine learning models?\nTerry McCann offers his helping hand.\nCalling all #DataScientists \u0026amp; #Engineers. I need your help! I am really interested helping data scientists learn how to deploy #MachineLearning. I am looking to chat to as many people as I can about how you #deploy models to support an article I am writing. (Please retweet) pic.twitter.com/jQLZ71os5q\n— Terry McCann (@SQLShark) 6 November 2018\r","date":"2018-11-11T15:00:53Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/11/last-week-reading-2018-11-11/","title":"Last week reading (2018-11-11)"},{"content":"It was very long and hard-working week (8.10-12.10). I was with my data-geek #SQLfamily and friends at the (SQL) Data Relay conference.\nThis is kind of log of that journey. Do expect various past and present tenses because I have written it during the event. Or wrote it icon-smile-o\nOld-New conference You\u0026rsquo;ve heard right: Data Relay. As a speaker I knew it before the conference began, but we have been asked to keep quiet until KeyNote time. Hence, SQL Relay has been transformed into Data Relay on 8/10/2018. Why? Simple reason. All the tools within the Microsoft Data Platform are not just about SQL-things anymore. There is Azure, PowerShell, Power BI, Azure Data Factory, Azure Data Analytics, U-SQL, Azure Databricks, Python, R, Spark. Even Java! Those days, old, worthy SQL is only a part of whole ecosystem we call Microsoft Data Platform. So, it\u0026rsquo;s what time to change the name and replace \u0026ldquo;SQL\u0026rdquo; with \u0026ldquo;Data\u0026rdquo;. Actually, our German friends have announced the same change recently.\nMy Role I was at the conference for the entire week this year. Regardless weather I was speaking or not. If not - I was a volunteer. Data Relay just like SQL Saturday is a free event. No one pays for tickets, no one raises money. Such events needs volunteers. There are a lot of things that must be done before and after each day of conference at every venue in every city.\nI have submitted a few session months before and happily, I was selected to provide one topic in two cities: Newcastle (Monday), Leeds (Tuesday). Demo and slides in my GitHub repository.\nFor the rest of the week, I was helping as a volunteer to support and see how the conference runs behind the scenes.\nMonday: Newcastle I arrived in Newcastle on Sunday evening. I\u0026rsquo;ve seen only part of the city but I like it already. I must come back soon. Quick sleep at Hilton Hotel (I do not recommend that one as the standard of room was below my expectations for Hilton). In the morning I ate the most expensive breakfast in my life as £20 for scrabble-eggs is so far ridiculous.\nThen we started preparing the venue. I had a session that day, but having I was in the last slot - so had plenty of time to (polish) and double-check the presentation. I had about 30 attendees in my session, which is not huge a number, but what was even more important for me - was the fact that they (almost) all had never used SSDT.\nAfter my session was the raffle and after that we packed up any remaining stuff into the bus heading to next stop: Leeds.\nFor the dinner, I joined the rest of the team and we went out to Fazenda. Paul [T] advised me about the huge quantities of meat severed at restaurant. I was hungry and it all looked very good. Paul told me they serve around 20 different types of grilled meat, so basically, you should not eat anything 24h before and you\u0026rsquo;ll not be able to eat 24 hours after feast.\nHe was right. After you put some salads onto a plate - the waiters were coming to you constantly and they were cutting or putting portions of specific meat directly onto your plate. Do I have to say that the meat was absolutely incredibly amazing?\nTuesday: Leeds Like yesterday - my session was in the last slot. That day I had full room of people (circa 60) and the topic was new for more of them. They have many questions during the session as well as after it. Everything went smoothly, attendees ware very happy - so me too.\nTime to get in to the bus and head to Birmingham. I can truly recommend a hotel where I stayed at, which is: Park Regis Birmingham. Very kind service, nice restaurant area and good breakfast and more important - a very comfortable room with big table to work at and a big bathroom. Ahh\u0026hellip; almost forgot - view from 12nd floor was beautiful, especially at 7 am when sun is rising.\nDuring the day I have saw the following sessions:\nStijn Wynants - Migrating to Azure Managed Instances! Barney Lawrence - Temporal Tables in the Data Warehouse Marek Jablonski - Azure Database Performance Craig Porteous - PowerBI and PowerShell. A match made in Heaven Wednesday: Birmingham I’m waiting for my next guest and we will be recording future episode of @SQLPlayer podcast, here in #DataRelay venue in Birmingham. Today\u0026rsquo;s venue is located in (centre) of Birmingham - at the ODEON Luxe Cinema. This contains 12 rooms (screens) where 4 of them were pretty big. I\u0026rsquo;m sad I didn\u0026rsquo;t get to present here - it would have been quiet a nice experience. The speakers haven\u0026rsquo;t even use ZoomIt during their presentations. Working as a helper I don\u0026rsquo;t have to present any session. Anyway, I confirmed an interview with a new guest of the \u0026ldquo;Ask SQL Family\u0026rdquo; podcast. The new episode will be recorded this evening. I can\u0026rsquo;t tell you with who, obviously.\nOhhh\u0026hellip; The rooms in the cinema were wonderful for another reason: for me it was first time I\u0026rsquo;d seen such comfortable chairs in a cinema. Fully leather, folding table and fully reclining, I could have easily fallen asleep in there.\nAfter the closing session we got onto the bus and headed to Reading.\nDuring the day I have saw the following sessions:\nRichard Munn - Patterns and Best Practices in SSIS Arun Sirpal - A Whirlwind Tour Of Azure SQLDB Simon Whiteley - Python Pipeline Primer: Data Engineering with Azure DataBricks Paul Andrew - Azure Data Factory v2 – SSIS Data Flows \u0026amp; Custom Extensibility Thursday: Reading We had a nice venue in Reading which was a Microsoft building in Thames Valley Park.\nAfter lunch I had a chat with another #SQLFamily member and you\u0026rsquo;ll see this in a podcast in the next few months.\nDuring the day I have saw the following sessions:\nTracy Boggiano - Intro to Query Store Abhishek Narain - Building your Big Data and Advanced Analytics Pipeline on Azure using Azure Data Factory Andrew Pruski - SQL Containers in the Cloud! An intro to running Docker in Azure We had good time in the bus in the way to Bristol. Having beer and chatting with friends - time flies.\nAfter we checked into The Bristol Hotel - we went out to Za-Za-Bazaar (all you can eat world buffet restaurant) for dinner. What a place! In this single restaurant you all the food you can imagine choosing from - you can eat whatever you want and only paying a fixed price for everything (excluding drinks).\nFriday: Bristol Today our venue for the conference was The Bristol Hotel - the same as where we stayed at. Some AV issues at the beginning didn\u0026rsquo;t stop us from a great event and final day.\nThank you all attendees, speakers and organizers. #sqlfamily see you next year!\nDuring the day I have saw the following sessions:\nAnnette Allen - Developing SQL Code to keep your DBA happy Alex Yates \u0026amp; Steph Locke - Live debate: Is data ops a thing? Pictures [srizonfbalbum id=11]\n","date":"2018-11-07T06:40:53Z","image":"/uploads/2018/10/Fazenda_IMG_4036_s.jpg","permalink":"/2018/11/sql-relay-became-data-relay-2018/","title":"SQL Relay became Data Relay 2018"},{"content":"Good morning folks!\nI wish you very good week at work, at home or wherever you are/will be. Last week brought super-exciting news to me. What it was? Check at the bottom. In the meantime - bunch of news:\nWhat is Azure Data Factory Data Flow?\nYou should find all the answers about the coming feature in ADF.\nIn this video Mark (T) presents how to join two sources and how to create new branch (stream of data).\nMigration from SQL Server to Azure SQL Database Managed Instance\nThis post shows you step by step how you can migrate on-premise databases to a new Azure SQL Database Managed Instance.\nWhitepaper: 4 pillars of SQL Server Monitoring\nGood reading about monitoring SQL Server. Written by 5 SQL Server experts.\nMicrosoft Future Decoded\nCatch up on all the news and announcements from the two days.\nSQL Server 2014 Service Pack 3 is now Available!!!\nTake a look the detailed list of improvements introduced in this SP3.\nPower Query M Primer (Part 10): Types—List, Record\nThat and 9 previous parts in the series let you understand M much better.\nWhat was the fantastic news for me last week?\nI was awarded MVP Data Platform!!! I\u0026rsquo;m really honoured and proud.\nI’m really thrilled and I’d like to spread this great news out that I received today. I’m really happy and proud of become MVP. Thank you Microsoft and all people who made this happen. It’s also a big responsible to carrying on what I was doing so far much stronger. #mvpbuzz pic.twitter.com/HoEtOusrON\n— Kamil Nowinski (@NowinskiK) 1 November 2018\r","date":"2018-11-04T23:00:51Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/11/last-week-reading-2018-11-04/","title":"Last week reading (2018-11-04)"},{"content":"In this post, I will explain what the DACPAC is and how to create it.\nIn the previous post in DevOps series, I described how to create a new, empty database project in Visual Studio.\nOne of the ways how to import an existing database (from a server) was import database from DACPAC file.\nThe question is:\nWhat the DACPAC is? DACPAC = Data Tier AppliCation Package\nDACPAC is a single file which contains database model i.e. all files represent database objects. It\u0026rsquo;s a binary representation of database project compatible with SSDT.\nThe name comes from an extension of these files. But, if you change the extension from *.dacpac to *.zip, you will be able to jump into and check what is over there.\nThis is an example of my file:\nContent of a DACPAC file Hence, DACPAC file contains all the information about model of a database needs to restore it. Let\u0026rsquo;s take a look inside:\n[Content_Types].xml\n\u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;utf-8\u0026#34;?\u0026gt; \u0026lt;Types xmlns=\u0026#34;http://schemas.openxmlformats.org/package/2006/content-types\u0026#34;\u0026gt; \u0026lt;Default Extension=\u0026#34;xml\u0026#34; ContentType=\u0026#34;text/xml\u0026#34; /\u0026gt; \u0026lt;/Types\u0026gt; DacMetadata.xml\n\u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;utf-8\u0026#34;?\u0026gt; \u0026lt;DacType xmlns=\u0026#34;http://schemas.microsoft.com/sqlserver/dac/Serialization/2012/02\u0026#34;\u0026gt; \u0026lt;Name\u0026gt;CRM\u0026lt;/Name\u0026gt; \u0026lt;Version\u0026gt;1.0.0.0\u0026lt;/Version\u0026gt; \u0026lt;/DacType\u0026gt; Origin.xml\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 \u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;utf-8\u0026#34;?\u0026gt; \u0026lt;DacOrigin xmlns=\u0026#34;http://schemas.microsoft.com/sqlserver/dac/Serialization/2012/02\u0026#34;\u0026gt; \u0026lt;PackageProperties\u0026gt; \u0026lt;Version\u0026gt;3.0.0.0\u0026lt;/Version\u0026gt; \u0026lt;ContainsExportedData\u0026gt;false\u0026lt;/ContainsExportedData\u0026gt; \u0026lt;StreamVersions\u0026gt; \u0026lt;Version StreamName=\u0026#34;Data\u0026#34;\u0026gt;2.0.0.0\u0026lt;/Version\u0026gt; \u0026lt;Version StreamName=\u0026#34;DeploymentContributors\u0026#34;\u0026gt;1.0.0.0\u0026lt;/Version\u0026gt; \u0026lt;/StreamVersions\u0026gt; \u0026lt;/PackageProperties\u0026gt; \u0026lt;Operation\u0026gt; \u0026lt;Identity\u0026gt;95014e0f-3cbe-440d-b780-e6dc8912b8b3\u0026lt;/Identity\u0026gt; \u0026lt;Start\u0026gt;2018-10-25T23:28:54.3649692+01:00\u0026lt;/Start\u0026gt; \u0026lt;End\u0026gt;0001-01-01T00:00:00.0000000\u0026lt;/End\u0026gt; \u0026lt;ProductName\u0026gt;Microsoft.SqlServer.Dac, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\u0026lt;/ProductName\u0026gt; \u0026lt;ProductVersion\u0026gt;14.0.4079.2\u0026lt;/ProductVersion\u0026gt; \u0026lt;ProductSchema\u0026gt;http://schemas.microsoft.com/sqlserver/dac/Serialization/2012/02\u0026lt;/ProductSchema\u0026gt; \u0026lt;/Operation\u0026gt; \u0026lt;Server\u0026gt; \u0026lt;ServerVersion\u0026gt;Microsoft SQL Server 2016 (SP1-GDR) (KB4458842) - 13.0.4224.16 (X64) Aug 18 2018 09:00:06 Copyright (c) Microsoft Corporation Enterprise Edition (64-bit) on Windows 10 Enterprise 6.3 \u0026lt;X64\u0026gt; (Build 17134: ) (Hypervisor) \u0026lt;/ServerVersion\u0026gt; \u0026lt;ObjectCounts\u0026gt; \u0026lt;ComputedColumn\u0026gt;16\u0026lt;/ComputedColumn\u0026gt; \u0026lt;DatabaseOptions\u0026gt;1\u0026lt;/DatabaseOptions\u0026gt; \u0026lt;Index\u0026gt;1\u0026lt;/Index\u0026gt; \u0026lt;PermissionStatement\u0026gt;2\u0026lt;/PermissionStatement\u0026gt; \u0026lt;PrimaryKeyConstraint\u0026gt;3\u0026lt;/PrimaryKeyConstraint\u0026gt; \u0026lt;Procedure\u0026gt;12\u0026lt;/Procedure\u0026gt; \u0026lt;Schema\u0026gt;1\u0026lt;/Schema\u0026gt; \u0026lt;SimpleColumn\u0026gt;27\u0026lt;/SimpleColumn\u0026gt; \u0026lt;SubroutineParameter\u0026gt;1\u0026lt;/SubroutineParameter\u0026gt; \u0026lt;Table\u0026gt;3\u0026lt;/Table\u0026gt; \u0026lt;/ObjectCounts\u0026gt; \u0026lt;/Server\u0026gt; \u0026lt;Checksums\u0026gt; \u0026lt;Checksum Uri=\u0026#34;/model.xml\u0026#34;\u0026gt;5C92D6C7DB735D17762884575B347ED9573AFF5C28FBE2A713313C620FAAAF47\u0026lt;/Checksum\u0026gt; \u0026lt;/Checksums\u0026gt; \u0026lt;ModelSchemaVersion\u0026gt;2.9\u0026lt;/ModelSchemaVersion\u0026gt; \u0026lt;/DacOrigin\u0026gt; model.xml\n\u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;utf-8\u0026#34;?\u0026gt; \u0026lt;DataSchemaModel FileFormatVersion=\u0026#34;1.2\u0026#34; SchemaVersion=\u0026#34;2.9\u0026#34; DspName=\u0026#34;Microsoft.Data.Tools.Schema.Sql.Sql130DatabaseSchemaProvider\u0026#34; CollationLcid=\u0026#34;1033\u0026#34; CollationCaseSensitive=\u0026#34;False\u0026#34; xmlns=\u0026#34;http://schemas.microsoft.com/sqlserver/dac/Serialization/2012/02\u0026#34;\u0026gt; \u0026lt;Model\u0026gt; \u0026lt;Element Type=\u0026#34;SqlDatabaseOptions\u0026#34; Disambiguator=\u0026#34;1\u0026#34;\u0026gt; \u0026lt;Property Name=\u0026#34;Collation\u0026#34; Value=\u0026#34;Latin1_General_CI_AS\u0026#34; /\u0026gt; \u0026lt;Property Name=\u0026#34;IsAnsiNullDefaultOn\u0026#34; Value=\u0026#34;True\u0026#34; /\u0026gt; \u0026lt;Property Name=\u0026#34;IsAnsiNullsOn\u0026#34; Value=\u0026#34;True\u0026#34; /\u0026gt; \u0026lt;Property Name=\u0026#34;IsAnsiWarningsOn\u0026#34; Value=\u0026#34;True\u0026#34; /\u0026gt; \u0026lt;Property Name=\u0026#34;IsArithAbortOn\u0026#34; Value=\u0026#34;True\u0026#34; /\u0026gt; \u0026lt;Property Name=\u0026#34;IsConcatNullYieldsNullOn\u0026#34; Value=\u0026#34;True\u0026#34; /\u0026gt; \u0026lt;Property Name=\u0026#34;IsTornPageProtectionOn\u0026#34; Value=\u0026#34;False\u0026#34; /\u0026gt; \u0026lt;Property Name=\u0026#34;IsFullTextEnabled\u0026#34; Value=\u0026#34;True\u0026#34; /\u0026gt; \u0026lt;Property Name=\u0026#34;PageVerifyMode\u0026#34; Value=\u0026#34;3\u0026#34; /\u0026gt; \u0026lt;Property Name=\u0026#34;QueryStoreStaleQueryThreshold\u0026#34; Value=\u0026#34;367\u0026#34; /\u0026gt; \u0026lt;Relationship Name=\u0026#34;DefaultFilegroup\u0026#34;\u0026gt; \u0026lt;Entry\u0026gt; \u0026lt;References ExternalSource=\u0026#34;BuiltIns\u0026#34; Name=\u0026#34;[PRIMARY]\u0026#34; /\u0026gt; \u0026lt;/Entry\u0026gt; \u0026lt;/Relationship\u0026gt; \u0026lt;/Element\u0026gt; \u0026lt;Element Type=\u0026#34;SqlSchema\u0026#34; Name=\u0026#34;[data]\u0026#34;\u0026gt; \u0026lt;Relationship Name=\u0026#34;Authorizer\u0026#34;\u0026gt; \u0026lt;Entry\u0026gt; \u0026lt;References ExternalSource=\u0026#34;BuiltIns\u0026#34; Name=\u0026#34;[dbo]\u0026#34; /\u0026gt; \u0026lt;/Entry\u0026gt; \u0026lt;/Relationship\u0026gt; \u0026lt;/Element\u0026gt; \u0026lt;Element Type=\u0026#34;SqlProcedure\u0026#34; Name=\u0026#34;[data].[PopulateAction_v1]\u0026#34;\u0026gt; \u0026lt;Property Name=\u0026#34;BodyScript\u0026#34;\u0026gt; \u0026lt;Value QuotedIdentifiers=\u0026#34;True\u0026#34; AnsiNulls=\u0026#34;True\u0026#34;\u0026gt;\u0026lt;![CDATA[ IF NOT EXISTS (SELECT 1 FROM [dbo].[Action]) BEGIN SET IDENTITY_INSERT [dbo].[Action] ON; INSERT INTO [dbo].[Action] ([ActionId], [ActionName], [UpdatedOn], [UpdatedBy]) icon-exclamation-triangle DACPAC does not contain DATA and other server-level objects. The file can contains all object types which might be kept in SSDT project.\nHow to create DACPAC file? There are few options:\nUse SSMS Use Visual Studio Use Visual Studio and create a snapshot of the database project (SSDT) Creating DACPAC using SSMS SQL Server Management Studio is a very popular and handy tool. You can generate DACPAC file using this tool, although I do not recommend it. Why? I will explain it in a minute.\nLet\u0026rsquo;s take a look at how to do that.\nOpen SSMS, right-click on database (in Object Explorer) and go to menu: Tasks -\u0026gt; Extract Data-tier Application\u0026hellip;\nSSMS: Tasks and Extract Data-tier Application Watch out. There are many options similar to each other. Use:\n- Extract Data-tier Application - to create DACPAC\n- Export Data-tier Application - to create BACPAC (more details in next post)\nThen you will be asked for the target location for the file.\nIf you are lucky - everything goes smoothly.\nIf not - you might encounter many errors hence the process fails generating no file.\nFor example:\nError SQL71561: Error validating element [dbo].[xyzView]: View: [dbo].[xyzView] has an unresolved reference to object [refDb].[dbo].[someTable].\nSSMS: When an extract DAC file goes wrong\u0026hellip; Predominantly, it\u0026rsquo;s no chance to fix the issues on the server and repeat the action, due to:\n* production environment - don\u0026rsquo;t touch anything without proper process of deployment\n* depends on number of issues - it might takes ages and you need your DACPAC now.\nThat\u0026rsquo;s why I do not recommend generating the file from SSMS.\nWhat other opportunity do we have?\nCreating DACPAC using Visual Studio This option is the most robust. Although a final file (DACPAC) will be the same at the end - Visual Studio offers more options than the predecessor. One of the more important one is an opportunity to turn off \u0026ldquo;Verify extraction\u0026rdquo;, which enabled (and hidden) in SSMS leads to failure in action. By disabling that option you will avoid the issue and it doesn\u0026rsquo;t matter whether the database contains errors or not - the DAC package file will be generated.\nOpen extract window by using menu: View -\u0026gt; SQL Server Object Explorer, select required database and right-click on it and choice Extract Data-tier Application\u0026hellip;\nVisual Studio: Extract Data-tier Application Then select all option as shown below:\nMore options with extract DAC in Visual Studio - SQL Server Data Tools Here, you have got much more options.\nExtract schema only Like previously - it\u0026rsquo;s generating a model only.\nExtract schema and data This option allows you to generate the data. Be careful! Unexpectedly your DACPAC file might grow to huge size and process can takes long time. Use this option wisely.\nWhat a minute\u0026hellip; data? The data, you said? Yes! But hold your horses. Personally, I have never used this option and apparently it has got limitations. One of them:\nExtract failed Extract failed. One or more unsupported elements were found in the schema used as part of a data package.\nError SQL71564: Foreign key Foreign Key: [Production].[FK_ProductProductPhoto_ProductPhoto_ProductPhotoID] cannot reference table Table: [Production].[ProductPhoto] when used as part of a data package.\nDACPAC file contains the data in BCP format Once the extract success - the DAC package file contains extra folders and BCP files as shown below:\nSelect tables\u0026hellip; Basically, not only tables but other objects like views, stored procedures, etc. This option is only available when \u0026ldquo;Extract schema and data\u0026rdquo; is chosen.\nAll objects are grouped by schema:\nVisual Studio: Choose the tables to extract data Verify extraction The option validates objects in extracted DAC package and you could end up with the same errors like in SSMS which, as a result, would not generate the file for you. It\u0026rsquo;s not what we expect. Especially, when planning to use the extract for import to Visual Studio as a database project - I do recommend verifying all objects over there.\nCreating snapshot of database project (VS/SSDT) SSDT: Create Snapshot Project You might want to create a snapshot of database model at specific point in time. It\u0026rsquo;s convenient method for keeping history of releases when your company haven\u0026rsquo;t introduced fully automated DevOps processes yet. Please, do not confuse it with database snapshot. This is different one. As the outcome you have got a DACPAC file of current version of database model. Simply right-click on database project (in Visual Studio) and choose \u0026ldquo;Snapshot Project\u0026rdquo;.\nIt will create \u0026ldquo;Snapshots\u0026rdquo; folder for you (if you haven\u0026rsquo;t yet) and save DACPAC file with a timestamp.\nImport As I described in the previous post (New DB project with SSDT and import existing database) in this series, usage the DACPAC file is one of the methods of importing objects to an empty, SSDT database project. Normally, a DAC package shouldn\u0026rsquo;t contain the data in it. And even though extracting from Visual Studio gives us that opportunity - you can\u0026rsquo;t import such file to database project. That attempt ends up with message:\nThis operation is not supported for packages containing data Summary Data Tier Application (DAC) is very powerful tool. You can create a new database project by importing from file. Having that you can use the DACPAC file in order to deploy changes onto target database server, compare changes between the file and another file or physical database. What\u0026rsquo;s more important: you can create whole CI/CD process. Believe me or not - since I have found the way of keeping database code - I can not image management without it. Also, I completely understand the pain of those who have to manage code of MySql or Postgesql database. They don\u0026rsquo;t have counterpart of DACPAC.\nThanks for reading and please leave a comment if you have any questions or own experiences with DACPAC files.\nUpdated @ 08/11/2019\nThis post is already part of the whole series about SSDT.\n","date":"2018-10-31T07:15:35Z","image":"/uploads/2018/10/honey-extraction-523918_cut.jpg","permalink":"/2018/10/how-to-create-dacpac-file/","title":"How to create DACPAC file?"},{"content":"Good day SQL folks!\nIt is Monday again\u0026hellip; I have no idea what can I say on this cold and cloudy day. Just take your time and read what I have prepared for you.\nWhat is Public Key Cryptography?\nLet\u0026rsquo;s start with the security first. A very good place to start with reading about securtiy. There are some code as well and a reference to my second favourite topic whic is ECC (Eliptic-Curve Cryptography)\nWhat is Azure Data Studio?\nMeet Azure Data Studio. Previously released under the preview name SQL Operations Studio, is a cross-platform database tool for data professionals using the Microsoft family of on-premises and cloud data platforms on Windows, MacOS, and Linux.\nData Is The Foundation For Artificial Intelligence And Machine Learning\nThis time we look into the Forbes magazine to find out that the most important thing in the future will be data, machine learning and AI. Good for us!\nOn-premises data gateway October update is now available\nThe October update for the On-premises data gateway has been released! I am very happy to see a support to the SAP-HANA single sign-on (in preview). Check what\u0026rsquo;s more this month!\nString or Binary data would be truncated: replacing the infamous error 8152\nYes, we have it! This message has been finally addressed in the SQL Server 2019 CTP2 but wait for it! You shoud expect it to be present in the SQL Server 2016 and 2017 :) Read the article for more details.\nAnd!!!! Don\u0026rsquo;t miss it!\nThere is a podcast with Itzik Ben-Gan published! And more to come very soon !\nCheers,\nDamian\n","date":"2018-10-29T01:00:53Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/10/last-week-reading-2018-10-28/","title":"Last week reading (2018-10-28)"},{"content":" Introduction Kalen Delaney has been working with SQL Server since 1987 when she joined the Sybase Corporation in Berkeley, California. Kalen has an independent international trainer and consultant since 1992. She has been a SQL Server MVP since 1993 and has been writing about SQL Server almost as long. Kalen has spoken at dozens of technical conferences, including every PASS conference in the US and multiple TechEd events in both the US and Europe. Kalen is the author or co-author of many books on SQL Server, including the series about SQL Server Internals.\nThis talk has taken place during SQLDay conference in Wroclaw (Poland), on 16th May 2018 (Wednesday).\nInterviewers: Kamil Nowinski \u0026amp; Damian Widera.\nWhy we have the #SQLFamily so great?\nHow long did take Kalen to update the book about SQL Server 7.0? What is Kalen\u0026rsquo;s advice about starting in IT market?\nWho did thanks to Kalen more times in the last three weeks than in the last few years? What kind of things Kalen loves to repair?\nDo you know the story about @sqlqueen twitter handle? You\u0026rsquo;ll find all answers out over here.\nTranscript Damian Widera: Would you like to introduce yourself and what you are doing?\nKalen Delaney: What am I doing\u0026hellip;\nDW: For business.\nKD: In the world or here?\nKamil Nowinski: Both.\nKD: Okay. Hello. I\u0026rsquo;m Kalen Delaney. I\u0026rsquo;m here in Wrocław.\nKN: Perfect!\nKD: I have to not remember how it\u0026rsquo;s spelt. If I try to look at the word then it\u0026rsquo;s totally confusing. But if I just remember the sounds - Wrocław\u0026hellip; I\u0026rsquo;m here in Wrocław for the SQLDay conference, which just ended. Wonderful, wonderful event. I was very impressed by the dedication of all the volunteers and the attendees. So many people, so excited about SQL Server, that\u0026rsquo;s just so wonderful to see. I\u0026rsquo;ve been working with SQL Server for a long time and I\u0026rsquo;ve been to a lot of conferences and yes, there\u0026rsquo;s always excitement there, but to come to Eastern Europe here and have so many people so excited also, it was just very unexpected. So very nice to see and I can see it just growing and growing, so that\u0026rsquo;s wonderful. Most of the work that I do is training, so speaking at a conference is just a really natural thing for me to just share what I want to share about SQL Server and it\u0026rsquo;s even better than my classes because I just pick and choose what my favourite topics are, just what I like to talk about. Also, sometimes I prepared my classes by using pre-cons. In fact, the original five-day class that I developed 10 years ago was taken just from a bunch of pre-cons that I had done over the years on various topics, and I just put them all together and I did a little bit of modification, of course, to make them fit and flow. But that\u0026rsquo;s how my first class started, was just a combination of several pre-cons. So I liked doing the pre-cons, it allows me to test out new material and see how people react to it, see how long it takes and how difficult it is to present it. If it is understandable. So it\u0026rsquo;s a really nice thing for me to be able to do. Most of what I\u0026rsquo;m doing is teaching, writing. I\u0026rsquo;m working on some white papers on Azure for Microsoft now and also I am partnering with Brent Ozar. I\u0026rsquo;m going to be doing some online training in June, just a month from now.\nKN: He was our first guest basically.\nKD: Brent was?\nKN: Exactly one year ago.\nDW: Last year here, and he was our first guest in\u0026hellip;\nKD: Yes, I remember that he was here and there was some confusion because I was talking to him and we both said we were going to Poland and the conferences were very close together on the calendar. So, we thought we were going to be at the same conference and then it was just like a few days before that we realized, \u0026ldquo;wait a minute, I\u0026rsquo;m going to this city and you\u0026rsquo;re going to that city and\u0026hellip;\u0026rdquo;\nKN: What happened?\nKD: How does this work? Oh, it\u0026rsquo;s two different conferences. So, yes, I didn\u0026rsquo;t get to see Brent. I haven\u0026rsquo;t actually seen him in person in quite a while, but I\u0026rsquo;m glad you got to talk to him.\nDW: So how did you start working with SQL Server?\nKD: Or with SQL?\nDW: or with SQL, ok.\nKD: SQL? I was at the university, I was in graduate school studying computer science and one of my professors was Michael Stonebraker who went on to start the Ingres company, the Ingres database system. He was a professor at UC Berkeley, where I was getting my graduate degree and I took a database class from him. And\u0026hellip; although, I will confess I had some problems with professor Stonebraker. He was not my favourite person in the world, but I was fascinated by databases and what you could do with them and retrieving data out of them. We didn\u0026rsquo;t do anything with performance tuning, it was all just storing the data and querying the data. But even that was just fascinating. So when I was ready to look for a real job out of the\u0026hellip; oh no. After I got my degree, I actually started teaching at the university for a while. I didn\u0026rsquo;t teach databases. This was at UC Berkeley and they had people like Michael Stonebraker to teach the database classes. I taught introductory programming for quite a few years, but then I needed a full time, better paying job than at the university and looked around. I was in Berkeley, California where the university was and there was a new little company that a friend of mine recommended, called Sybase.\nKN: Oh, that\u0026rsquo;s how the history started.\nKD: So I went and interviewed and they were hiring like crazy. I interviewed on a Thursday and they called me on Monday morning saying, when can you start? Well, how about Wednesday? I started two days later, less than a week after my interview. Started working for Sybase and had no idea what I was getting into, but that\u0026rsquo;s how it all started.\nKN: Amazing history.\nKD: It was over 30 years ago. I know the date that I started because after one year, Sybase would give you a certificate that said \u0026ldquo;I survived a year at Sybase\u0026rdquo; and I had the certificate up on my wall for many years and so it had the date I started on - October 27th, 1987. So 2017 in October was 30 years and it just turned out to be really lucky for me that Sybase ended up partnering with Microsoft or that Microsoft chose Sybase to partner with because it was just such an easy transition after 5 years. My family then moved to the Seattle area where Microsoft was and I was able to just start working with their product very easily because it was the same product in those days. I never actually worked for Microsoft as an employee, like I was an employee for Sybase. But I started contract training for them almost immediately and then I started writing and working with them and to get to know the engineers that actually wrote the product at Microsoft was a very, very exciting thing. And I\u0026rsquo;m still excited about this product and I still like sharing about it and some day I might retire, but it keeps being exciting. So maybe not for a little while.\nKN: So you were working with the products but not as an employee of Microsoft?\nKD: No, never as a Microsoft employee.\nKN: But you were so close. Why did you decide to choose a different way? I mean, maybe not different, but why you didn\u0026rsquo;t choose to be a Microsoft employee?\nKD: You know, at the time when we moved up to the Seattle area, we decided we didn\u0026rsquo;t want to live right close to Microsoft. We left Berkeley, which was getting very, very crowded and very expensive. It wasn\u0026rsquo;t quite so expensive right near Microsoft, but it was still quite expensive. We wanted to live out in the countryside and I had small children and I didn\u0026rsquo;t want to have to drive two hours to work every day.\nKN: Absolutely.\nKD: So if we had had more money when we moved up there and could have afforded a house closer to Microsoft, I might\u0026rsquo;ve looked for a job with them, but we wanted to live out of the way and so I started doing consulting where I could not have to go in every day to an office. I could do a lot of work remotely or if I needed to teach a class, then I would just fly somewhere. But to have to drive two hours to work every day. I chose not to do that.\nDW: It\u0026rsquo;s just a waste of time, you can do anything else.\nKD: There were people, I did meet people who lived out near me who did commute into Microsoft every day, but I didn\u0026rsquo;t want to make that choice.\nKN: So basically training is in your blood.\nKD: I had been for a very long time. Before university I did some teaching assistance in high school. I was a teaching assistant for Math classes in high school, so very long time.\nKN: So I would say that you probably don\u0026rsquo;t need any special preparations for workshops or for pre-cons.\nKD: I\u0026rsquo;ve gotten some good advice from people over the years and I\u0026rsquo;ve seen people who I think are wonderful presenters that I\u0026rsquo;ve borrowed some of their techniques and tricks from. But no, I never did any specific training on how to be a presenter. So yeah, it just always seemed like something I just could do. I like to explain things.\nDW: Do you have any ideas how a person who would like to start to be a presenter, what the person should do, how to prepare, how to not to be feared on stage?\nKD: Not to have stage fright? You know, I don\u0026rsquo;t know if I have good advice for that. When I was teaching at the university, sometimes what I would do\u0026hellip; When I taught at the university, we had huge classes, sometimes 300-400 people teaching beginning programming to. This was Berkeley, everybody wanted to learn how to use computers at Berkeley in those days. And I would get the first day of class I would be very, very nervous, but what I would do is I would show up early and see all the other people that were waiting for the class to start. And sometimes I would just sit down with them and in the back of the room, like I was another student, and just see what the room looked like from the perspective of a student. And I could sense that the students were sometimes a little nervous too. This was the beginning computer science class so a lot of them had no idea what it was about. And they were scared if it was going to be too hard for them, if they were going to understand what was going on. And up until maybe five minutes before the class was to start, I would just try to feel like I was one of the students and then just a few minutes before we started, then I would get up and go out the back and come around and come up to the front of the room. And every once in a while there were a couple of people who recognized that I was the person sitting next to them, but usually not. And it just gave me more of a comfort because I knew both sides. I knew that these are all just people here, they\u0026rsquo;re waiting to see what they can learn here and I\u0026rsquo;m waiting to see how they\u0026rsquo;re going to accept what I have to offer them. And if I feel like I have something, I don\u0026rsquo;t have to be perfect, I don\u0026rsquo;t have to know everything, but I have something that I can offer them. I can share what I do have. And I never expected myself to be absolutely perfect. I just believed that I had something I could share. We\u0026rsquo;re all just people.\nKN: We used that question like: \u0026ldquo;are you a perfectionist\u0026rdquo; sometimes, during the interviews, so now we know that you are not going in that direction? Don\u0026rsquo;t have to be, okay?\nKN: So about work-life balance, I know that you chose to live outside of the city centre and that kind of way of life, but how is your work-life balance right now? How can you manage that, you know, a lot of flights across Europe, across the world basically? Your family and other private things\u0026hellip;\nKD: Now, it\u0026rsquo;s different because my children are all grown and one of my children lives in Europe so to come and fly to Europe to do a conference or a class is part of my life because then I get to see my daughter and my grandson who are in Germany, so that actually makes it easier.\nKN: So, you have an excuse [laughter].\nKD: Exactly. And I can see them more often than I would if I didn\u0026rsquo;t have this work. So that\u0026rsquo;s wonderful. When my children were littler, it was very difficult and sometimes I do wonder if I left them too much, but my husband was at home all the time. My husband was basically a full-time dad, so there was someone I was leaving the children with and I didn\u0026rsquo;t have to worry about who\u0026rsquo;s taking care of them. It was hard to be away, and I\u0026rsquo;m always, not always, but I sometimes get jealous of my colleagues now who have small children because we have this thing called Skype and when I see my colleagues say, \u0026ldquo;oh, just a minute, I\u0026rsquo;m skyping with my daughter\u0026rdquo; and it\u0026rsquo;s like a little three-year-old and they\u0026rsquo;re talking on Skype on their phone to a little three year old and they can see her and she can see her papa and I wish I had that when I first started travelling. It was a big deal to make an international phone call to call home and certainly no video, and just a phone call. So maybe I would do it two or three times in a week, but it wasn\u0026rsquo;t the same at all to have the video and to not have to pay lots of extra money just to make a call, it was very expensive. And for little children, just having audio was not as interactive for them. A lot of small children can\u0026rsquo;t do phone calls, but video, when they can see you, is very different and little children are much better with that. I have a grandson now who\u0026rsquo;s only one and a half and I have video calls with him now and he\u0026rsquo;s starting to recognize me just on Skype. Yeah. So I get it now with my grandson, but I feel bad that I wasn\u0026rsquo;t able to do that with my children.\nKN: But now you are happy if you have to go to Europe, right?\nKD: Oh, now going to Europe is fine. I have friends in many, many places and now that my children are grown, my friends are much more important to me. And so when I travel, I frequently make opportunities to go and have some time with my friends.\nDW: Do you remember your one of your first books? It was like the year 2000 or something. It was Black Bible, something like this, SQL Server something\u0026hellip; SQL Server 2000. And you were one of the authors.\nKD: There was \u0026ldquo;unleashed\u0026rdquo;. There was \u0026ldquo;SQL Server Unleashed\u0026rdquo;. And that was before 2000.\nDW: Something like this.\nKD: By 2000, I was already writing the internals books. I wrote the first Inside SQL Server for version 7, so that was like in 1997 or 1998.\nDW: I mentioned it because that was my first book. I started learning SQL and you were one of the authors\u0026hellip;\nKD: Yes, I did write a few chapters for SQL Server Unleashed for version 6 and 6.5 and maybe even for 7. But then I was asked to take the SQL Server, \u0026ldquo;Inside SQL Server\u0026rdquo; is what it was originally called. And someone who worked for Microsoft, Ron Soukup was a project manager at Microsoft. He had been given a leave from his job and allowed to take full time for nine months to write the first SQL Server book, to write the Inside SQL Server for 6.5. But even though he was paid his full salary and got to devote all his time to writing this book, he didn\u0026rsquo;t like it. He said he hated it and he never wanted to write another word again. So they offered me the possibility of updating it and there were so many changes for SQL 7. It was a major project. It took me another nine months, but I wasn\u0026rsquo;t being paid full-time salary to do that and I had some colleagues, a friend at Microsoft in fact, who tried to talk me out of it. He said this would just be too much work. It will just consume your whole life if you try to do this. And I remember we went out to dinner in Redmond and he was like doing everything he could to convince me this was not a good thing to do and I didn\u0026rsquo;t take his advice.\nDW: And that\u0026rsquo;s great because we have a really great book.\nKD: I don\u0026rsquo;t regret it. I do not regret it. There were some hard times and there were some many, many late nights, but I don\u0026rsquo;t regret it at all now. I\u0026rsquo;m very, very happy that I had that opportunity. I\u0026rsquo;m very glad that I got to\u0026hellip; And it was a door to meeting so many of the people at Microsoft, I mean all of the developers at Microsoft that I know personally and I can send an email to when I have questions and I met them while working on this book. And they were so helpful and so friendly and some of them are still good friends, and now I feel like I know people that I can talk to about whatever questions I have has made it a really wonderful opportunity.\nDW: It was also one of the first books I read entirely in English because the Polish translation\u0026hellip; translations of technical books into our language is sometimes not accurate.\nKD: I\u0026rsquo;ve heard that from other people.\nDW: I thought it would be better to just read it in English and learn something about also how to say something in English in technical language, to better understand.\nKD: It would help you with your, with your technical English.\nDW: I thought it wouldn\u0026rsquo;t be that easy to read, but then I found out it\u0026rsquo;s easier to read in English, to read the content in English because everything else was in English. Like we started having Internet those days and the technical documentation, the MSDN, which you could order on DVDs was also in English. I still have the original MSDN library.\nKD: I have stacks of MSDN disks still.\nDW: I have them as well, 7 disks. It was the entire MSDN of all products.\nKD: They created a new one every quarter or every few times a year. And I didn\u0026rsquo;t want to get rid of anything I might needed. I think I still have, I probably have 100 discs that I don\u0026rsquo;t know if they\u0026rsquo;ll ever be usable for anything, but it\u0026rsquo;s so hard to get rid of something like that.\nDW: Yes. It was great because when you were travelling for a business trip, you could have a disk on the plane or tram or wherever you were, you could have the library with you.\nKD: If your laptop had a CD player, and now they don\u0026rsquo;t\u0026hellip;\nDW: Not now, but at that time yes, we had them.\nKD: But now you can get things on USB and so you can pack everything you ever need to know on one little stick.\nKN: So you have a very wide field of observation due to the history and theory is changing right now and what is the direction of the changes around technology, around what Microsoft is doing and in which direction they are going to. So how do you feel about the direction? How do you feel about those changes? Like everything is in the cloud, Azure Cosmos DB, Azure Data Warehouse and all those things. What do you think about it?\nKD: Well, I don\u0026rsquo;t know all those things. I very much just stay with the engine but that of course is changing too and I have started doing work with Azure and it\u0026rsquo;s still just the engine but now it\u0026rsquo;s running on Azure. It\u0026rsquo;s running on someone else\u0026rsquo;s computer which is what the cloud is. And I know Microsoft is like betting the farm on the cloud and so far it seems like that it wasn\u0026rsquo;t a bad choice. At first, I was a little bit concerned because it was like anything anybody had to do had a cloud focus. They switched a lot of their internal consulting people to just pushing for cloud. And in fact, I was interviewing with Microsoft, I was looking for a full time position a little over a year ago and I did some consulting with Microsoft and it was very much focused on cloud and whatever I did, I would have to be able to do it in the cloud and work in the cloud and I ended up taking another job where I could do some cloud but still, there was still a lot of work to be done with customers running on-Prem.\nSo I think it\u0026rsquo;s a good choice to be\u0026hellip; It rings more like all the science-fiction stories that I\u0026rsquo;ve read, that information is always available from wherever you are. You just have a little watch that you can tap and ask any question, or from wherever you are, you can access your information or anyone else\u0026rsquo;s or any public information of course, rather than everybody having private storage and you have to be right there in order to access it. That seems like that is the way in the future. As far as all of the other technologies that Microsoft is introducing, I have not really dived into them. I really don\u0026rsquo;t have a lot of\u0026hellip; There\u0026rsquo;s too much. I\u0026rsquo;m trying to keep my field of vision a little more manageable.\nDW: So it\u0026rsquo;s too much only if you think about data platform products or components. And imagine the entire cloud, there are a lot more products like CRMs or Dynamics\u0026hellip;\nKD: Yeah, the job I had last year, they wanted me to get a certification for Amazon cloud and my brain just started spinning around all of the different services, all of the different components that you can plug into your AWS cloud, and I could not remember all of them. And then I had to take these little online exams and \u0026ldquo;You\u0026rsquo;re building an application that does this and this. Which components would be the ones that would help you the most and\u0026hellip;\u0026rdquo;\nDW: The key is \u0026ldquo;the most\u0026rdquo;, you can think in different ways, but\u0026hellip;\nKD: Yes, it was a good thing it was like open notes because I had to go all the way back and look and see \u0026ldquo;now, what was that one for?\u0026rdquo;, just too many to remember. And that was just for Amazon and Microsoft has its own and Google has a whole bunch of services and to try to keep track of who\u0026rsquo;s doing what and who does what better than who else -- that is not easier in the cloud. The idea of the cloud is good, but keeping track of everything there is no easier than keeping track of all the different pieces we have anywhere.\nKN: What do you think about the community around SQL Server?\nKD: I was just asking someone this the other day about \u0026ldquo;is there any other technology that has this kind of a community?\u0026rdquo; I mean the people use the hashtag #SQLfamily and I think it\u0026rsquo;s really true. I mean, there are people that have problems with each other, there are people who don\u0026rsquo;t like each other but that\u0026rsquo;s true in any family. But the fact that of people helping each other out and you know there is someone there who will help you or take care of you if you need it -- that\u0026rsquo;s also family. And a family is where you belong, and the fact that once you get into the SQL world, you\u0026rsquo;re just part of this family. I don\u0026rsquo;t know how this started, but it has been for a long time. When I first was an MVP, there were just a few of us and there was an MVP Summit, I believe the first one I noticed. This was in Dallas, we had our MVP Summit in Dallas and there were only maybe like less than 10 SQL Server MVPs.\nKN: Very beginning.\nKD: Very beginning and we all really liked each other and we hung out together all the time. And I did have a friend who was an Exchange MVP and he came over and talked to me at one point and he goes: \u0026ldquo;you know, some of the other Exchange MVPs are asking why do those SQL people hang out together all the time.\u0026rdquo; They thought it was strange. It\u0026rsquo;s like we didn\u0026rsquo;t like to go off on our own, we didn\u0026rsquo;t go meet other MVPs from other technologies. The SQL people liked to being with other SQL people.\nKN: Sounds like a perfect world.\nKD: And we all really liked each other and these are my friends and, you know, if I didn\u0026rsquo;t like anybody there, then I would go and find someone else to talk to. But I was perfectly happy just being with my friends who were the other SQL Server MVPs. And that started very long ago, and as more MVPs came in and then the PASS conference started, which obviously wasn\u0026rsquo;t just for MVPs, so other people got invited to just be there talking about SQL Server and sharing and hanging out together. Obviously, we don\u0026rsquo;t all hang out as one big 5,000 person group but I think the feeling of connection just didn\u0026rsquo;t disappear. And I don\u0026rsquo;t know of the reason why but it started that way and it\u0026rsquo;s just continued that way.\nDW: So SQLDays is also a community conference. It\u0026rsquo;s built by the community. It\u0026rsquo;s organized by the community. It\u0026rsquo;s the same situation.\nKD: So you have your family here in Poland but then you\u0026rsquo;re connected to the larger family as well.\nDW: We try to, as much as possible we try to do this.\nKD: There are people in your group who are definitely connected to the bigger group. So it\u0026rsquo;s all connected.\nKN: We were going to the conference across Europe as well so we have community members, SQL family.\nKD: All over Europe and then you come to the States\u0026hellip;\nDW: I go to India in August for DPS with Edwin [T] and some other people. Denny Cherry [T] will be there also.\nKD: Yes, he\u0026rsquo;s been there before.\nDW: Yes, he was. And also some other guys from the US but not only from the US, across the world.\nKD: India has a huge SQL community too, yes.\nDW: So it\u0026rsquo;s a big conference, like 5,000 people will come.\nKN: For India it\u0026rsquo;s a small conference.\nDW: It\u0026rsquo;s actually not the smallest one but it\u0026rsquo;s organized by Amit Bansal [T], who is also an MCM in SQL Server, so a famous person as well.\nKN: So what would you advise to a person who would like to start in IT market or maybe with SQL Server, working as a professional. Or how to start in IT, maybe in general?\nKD: I think going to a SQLSaturday would be a good place to start, and go to lots of different sessions to see what aspect of SQL Server you were interested in and then almost all the speakers will have sample scripts that you can try and you can get a developer edition of SQL Server for no charge and find what area of SQL Server you like and just start playing with it. And then after you feel like you know a little more, then go to another SQLSaturday and maybe talk to companies that are working with what you like doing and maybe some of them are hiring. So I think, because SQL itself is just so big, you need to find what within it you\u0026rsquo;re interested in, and to be able to go to SQLSaturday for no charge and start to get some exposure I think could be a wonderful thing to do.\nSo I\u0026rsquo;ll tell you another story then about\u0026hellip; I have four children but only one of them is really, really into technology. My middle son wants to be a game programmer but he\u0026rsquo;s just a brilliant programmer and he has a job doing software development, not games yet, but he keeps trying, keeps interviewing with games companies. But he\u0026rsquo;s doing development at his company and someone else in his company in IT is working with MySQL and knows that my son is so smart, and so has started coming to him for help with SQL, even though my son doesn\u0026rsquo;t really know it that well, he can help a little bit because he learns things instantly. But a few things he couldn\u0026rsquo;t answer, so he started asking me! So to have my son asking me SQL questions, is a real thrill. And in fact, he said \u0026ldquo;thank you\u0026rdquo; more times in the last three weeks than he was like in the last few years. It\u0026rsquo;s like he\u0026rsquo;ll ask me a question and I\u0026rsquo;ll answer and he\u0026rsquo;ll go, \u0026ldquo;oh, that\u0026rsquo;s perfect mom, thanks!\u0026rdquo; Wow, this is just wonderful to get from my son! And I have these little tiny thoughts that maybe he\u0026rsquo;ll get more interested in SQL and want to do SQL. But he\u0026rsquo;s such a wonderful developer, a full stack developer, and he does incredible things with graphics and he does music and he can write an entire game on his own. He does the graphics. He can do the graphics. He can do the music. He does the programming. He\u0026rsquo;s also an actor and so if it\u0026rsquo;s a game with voices, he actually did some professional voice acting for a while so he can add his own voices! So I think SQL would just be too limiting for him, but it is fun. It is really fun to just have him start asking relatively starting out questions about SQL. So sometimes that\u0026rsquo;s how people start their job doing one thing and they see that there are people doing SQL and maybe they just do it an hour or so and then\u0026hellip; I\u0026rsquo;m sure if my son wanted to do SQL at the job there, he might be able to find his way into it now that they know what he\u0026rsquo;s capable of, now that they know how smart he is and that he can learn. I don\u0026rsquo;t think he wants to do it, though. But other people who do want to do it, if they\u0026rsquo;re in a technical job or any job that has an IT department that probably has some SQL going on, that\u0026rsquo;s another place they could start to look. Is there something at their own company involving SQL that they could just start touching? So just one possibility.\nDW: I would like to ask you about your hobby. Do you have any other hobbies apart from SQL?\nKD: SQL is not my hobby. It is a passion. I feel very blessed that I do enjoy the work that I do, but sometimes I do non-SQL but when I\u0026rsquo;m working, it\u0026rsquo;s all SQL Server. I do needlework. I do knitting and crocheting and I just started making jewelry, mostly I just repair my own jewelry that broke, in fact just today I was wearing a necklace today and during the final session of this conference, I was turning and my hand grabbed it accidentally and it broke. But I think when I get home, I\u0026rsquo;ll be able to fix it. Because I have jewelry tools now. So I fix and make necklaces and earrings and I knit and I make things for my grandson. And I like to read.\nDW: It requires a lot of precision, I think, to repair jewelry?\nKD: Just better eyesight than I have, but I do have this big magnifying glass with a light attached to it. So it feels really, really professional. I bring this light over and then I can look down with my tools and fix what\u0026rsquo;s broken. Without that magnifier my eyes are not good enough anymore.\nKN: So where we can find you at the next events?\nKD: Next, mostly I\u0026rsquo;m writing white papers, working with Microsoft for most of my work for the next few months, but I will be at SQLSaturday in Sacramento in July. I will be online with Brent Ozar in June. I have a half-day class on locking and a two-day class on internals for query tuning, the things you need to know about indexes and the engine in order to do the tuning. It\u0026rsquo;s not about tuning, it\u0026rsquo;s the internals to help you become a better tuner. Those classes are both in June and then not travelling again until October when I have two classes in Europe. I have a class in Sweden and a class in Denmark in October. And these are all on my website, which is sqlserverinternals.com.\nDW: That was our next question.\nKN: Yes, exactly.\nKD: Or if you forget that, you can ask me on Twitter.\nDW: And that was our next question.\nKD: @sqlqueen and I do have to apologize because when I first joined Twitter, I didn\u0026rsquo;t think it would be so public. I joined Twitter because a colleague that I was doing a project with said, when I was having trouble getting a hold of him, I had questions about the next step of the project and I said: \u0026ldquo;how come you\u0026rsquo;re not answering your emails?\u0026rdquo; And he goes: \u0026ldquo;writes me a message on Twitter and I\u0026rsquo;ll always answer.\u0026rdquo; So I had to join Twitter and I said: \u0026ldquo;I don\u0026rsquo;t understand what this Twitter thing is and what do I have to do here. And I have to come up with a Twitter name.\u0026rdquo; And he just threw that out. \u0026ldquo;Oh, well, how about if you be sqlqueen?\u0026rdquo; And I just thought it would be for talking to him and not realizing that the whole world would start calling me SQLQueen.\nKN: I think he\u0026rsquo;s completely okay. I totally think and probably everyone could agree that you completely deserve that title.\nKD: For me to come up with that on my own, usually I don\u0026rsquo;t do that. When I start a company, I am not somebody who would name my company after myself, so to have a Twitter handle that just seems to be aggrandizing normally would not be something I would do. But I thought it would just be a private thing with this one colleague and I didn\u0026rsquo;t know it would grow!\nDW: It\u0026rsquo;s very easy to remember.\nKD: And I don\u0026rsquo;t mind it so much, but it isn\u0026rsquo;t what I would have thought of if I knew what Twitter would become and that I would be doing so much communication on Twitter.\nDW: Actually, it helped us when we announced this conference last year, we also announced that you will see the real SQL Queen.\nKD: Okay. Well, I\u0026rsquo;m glad it was useful to you!\nKN: Thank you for accepting our invitation to this interview. So basically we can say that finally it has happened!\nDW: After months! 7 months, I think.\nKD: How many times have we tried\u0026hellip;\nDW: It was worth waiting for this moment.\nKN: Especially to meet you here in Wrocław, in Poland here. Great, thank you very much.\nKD: Thank you. Good to talk to you.\nUseful links Kalen\u0026rsquo;s twitter: @SQLQueen\nKalen\u0026rsquo;s website: SQL Server Internals\nTrainings - Available Classes\n","date":"2018-10-22T23:14:53Z","image":"/uploads/2018/10/ASF015_SQLDay_with_Kalen_Delaney_.jpg","permalink":"/2018/10/asf-015-kalen-delaney-interview/","title":"ASF 015: Kalen Delaney interview"},{"content":"DevOps Terms – and Why They Matter to Database Specialists\nFind out what is Scrum, Continuous Deployment, TDD and others from Kendra Little [T].\nSeamless access to management insights for SQL Data Warehouse with Data Studio\nCheck potential physical data skew in your Azure SQL Data Warehouse (SQL DW) using Azure Data Studio (general availability)\nSQL Server 2019 CTP 2.0 New Features – Introducing the Page Cracker (AKA sys.dm_db_page_info)!\nThat new function (fn_PageResCracker) helps you decoding a page identifier. No more DBCC PAGE usage needed.\nAggregations for petabyte-scale BI available in the Power BI service\nDo you know all the good reasons to use Aggregations (which is in public preview at the moment)? Find out them and watch a demo (video) plus deep dive demo from @Guyinacube [T].\nPowerBI – Constant Line\nQuick one about how to add it to a chart.\nT-SQL TO U-SQL DATA TYPE CONVERSION\nThat is useful (although not sure how long yet) cheat sheet table from Paul [T].\nData science in SQL Server: pivoting and transposing data\nDejan Sarka presents the same trick but in R.\nData models within Azure Analysis Services and Power BI\nBest practices useful particularly when creating data semantic models in SQL Server Analysis Services, Azure Analysis Services, or Power BI.\nData preparation with Debug mode in Azure Data Factory v2 [T]\n","date":"2018-10-21T22:12:51Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/10/last-week-reading-2018-10-21/","title":"Last week reading (2018-10-21)"},{"content":"Good day SQL folks! The conference season has begun. You can find very interesting conferences every week in almost every place in the globe. Of course you do not need to find that far. Most of the time you can find great meetings just near by you.\nSQLPlayer was one of the partner of such event that took place in Wrocław on 6th October. It was BI Connected by SII conference which was dedicated to the modern BI solutions.\nI have met many good friends from our DataCommunity User Group (Roman). It was a great pleasure to hear to great talks. Everyone could find something interesting just to learn something new. I was very interested in the ETL topics (Maciej Grzybowski) and also about live reporting using Qlick Sense. Especially when people were sending cats (Jacek Harazin). People were very keen hearing about the analytics in SQL Server using Python and/or R (Łukasz Grala). Of course that was not all as Adrian Chodkowski explained one of the hottest topics in SQL Server - the in-memory engine. Plus lot of in information about Microsoft Azure and the support to the BI operations (Leszek Kwaśniewski \u0026amp; Borys Bykowski).\nI would like to thank to the organizers for preparing a very professional event. It is always a good idea to share knowledge and that was one of the event I would like to be next time as well!\nCheers\nDamian\n","date":"2018-10-17T01:18:03Z","image":"/uploads/2018/09/BI-Connected-logo.jpg","permalink":"/2018/10/bi-connected-by-sii-conference-in-wroclaw/","title":"BI Connected by SII - conference in Wrocław"},{"content":"\u0026ldquo;This was in Dallas, we had our MVP Summit in Dallas and there were only maybe like less than 10 SQL Server MVPs.\u0026rdquo;\nDo you want to know who has said that and why we have the #SQLFamily so great?\nWatch the video trailer and guess who is the next guest.\n","date":"2018-10-15T23:19:10Z","image":"/uploads/2018/10/ASF-015-video-invitation.jpg","permalink":"/2018/10/asf-015-video-invitation/","title":"ASF 015 video invitation"},{"content":"Hi guys! Not too much this week as I was very busy on #DataRelay tour across UK whole week (relation soon).\nAzure Data Factory - Data Flow Monitoring\nCheck what you can do in terms of monitoring and execution plan in ADF.\n[video]\nGet cloud-scale analytics of Office 365 data with Azure Data Factory\nGaurav Malhotra presents how you can use Azure to get innovative analytics and insights from Office 365 data in a way that was not possible before.\nGroupBy voting\nGroupBy is free online conference where you pick the lineup. Vote now.\nReader solutions to Special Islands challenge\nHandy pre-2012 compatible version of LAG/LEAD-like calculations.\nPower BI Developer community September update\nIn case you missed it.\nSSMS 18.0 public preview released\nThis new version is based on the new VS 2017 Isolated Shell.\n","date":"2018-10-14T23:25:27Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/10/last-week-reading-2018-10-14/","title":"Last week reading (2018-10-14)"},{"content":"Good day SQL folks!\nIt is Monday again\u0026hellip; I have no idea what can I say on this cold and cloudy day. Just take your time and read what I have prepared for you.\nFun with DAX – Minesweeper\nSo y\u0026rsquo;all bored on Monday? Check out the contest posted by Philip Seamark. If you like the good old Minesweeper game and love playing with Power BI than this is just for you.\nSyllabuck: Ignite 2018 Conference\nWhat can I say - all Ignite sessions in one place. Great job Buck Woody! You can go directly to YouTube channel looking for great sessions!\nSummary of my Container Series\nThis is link to collection of blogpost that Andrew Pruski has written about containers. I got to work a bit with containers and his posts are great introduction to this topic.\nWhat to do (or not do) about top wait stats\nAaron Bertrand discusses waits\u0026hellip; I love his blog and the approach he has to the topic. Not only to this one but in general. It is always worth to spend time reading his articles. By the way - Aaron will be speaking during the SQLDay 2019 !\nJoe Sack discusses the new feature of the SQL Server 2019. Great! Waiting for more demos like this one!\nBonus - Colossal Collection of Convenient Cheat Sheets for Curious DataScientists\nBecause sometimes we need just to have a quick look on the material\u0026hellip;.\nAnd!!!! Don\u0026rsquo;t miss it!\nThere is a podcast with Itzik Ben-Gan published!\nCheers,\nDamian\n","date":"2018-10-08T11:32:41Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/10/last-week-reading-2018-10-08/","title":"Last week reading (2018-10-07)"},{"content":"In this blog post I want to quick go through one of useful capabilities that Microsoft provided with version 2 of Azure Data Factory.\nAs a developer I always want to use code repository to keep all my changes, to manage tasks, branches, share the code with a team and simply\u0026hellip; keep it in safe place. That works very well for many years for various code of applications as well as database projects (SSDT). But how to use code repository for tools who shares UI via browser as ADFv2 does? Doing a copy of exported ARM templates and committing them manually to code repository isn\u0026rsquo;t efficient, repeatable and no-errors burdened process, after all.\nAttach project to Git repository We can attach a code repository to the Azure Data Factory v2. You can do that for existing factory or for new one and let\u0026rsquo;s assume that you already have got a Git repository. Microsoft supports Git in Azure DevOps (formerly VSTS) only as GitHub repositories would be enabled in GA version of factory.\nYou can start doing it from two places:\n1) Main dashboard - \u0026ldquo;Overview\u0026rdquo;. Click \u0026ldquo;Set up Code Repository\u0026rdquo; button (first from the right)\n2) Go to design mode by selecting \u0026ldquo;Author\u0026rdquo; button and click \u0026ldquo;Data Factory\u0026rdquo; \u0026gt; \u0026ldquo;Set up Code Repository\u0026rdquo; (top-left corner)\nADFv2 - Overview dashboard Set up Code Repository from Author section When we choose \u0026ldquo;Azure DevOps Git\u0026rdquo; in Repository Type - the list of accounts available for you will be automatically filled up:\nRepository Settings - Azure DevOps Account step Then you will see all the project inside the project account you have just selected in the field \u0026ldquo;Project Name\u0026rdquo;. In this case I\u0026rsquo;ve chosen \u0026ldquo;DataServices\u0026rdquo;:\nRepository Settings - Project name step Subsequently you must select Git repository name. In this step you can create new repository or use an existing one. In this case, I used an existing repository called \u0026ldquo;RepoTest1\u0026rdquo;. Having that, do point out to \u0026ldquo;Collaboration branch\u0026rdquo; which is your Azure repository collaboration branch that is used for publishing. By default it is \u0026ldquo;master\u0026rdquo;, but you can change it if you want to publish resources from other branch.\nIn \u0026ldquo;Root folder\u0026rdquo; you can put the path which will be used to locate all resources of your Azure Data Factory v2, i.e. pipelines, datasets, connections, etc. Leave it as is or specify if you have more components/parts in the project\u0026rsquo;s repository.\nImport existing Data Factory resources to repository That option allows you to make a initial commit of your current ADF into selected branch in the repository. Select the option when your repository has just been created or target folder is empty. Otherwise, you will get started with an empty project (synced to repo) whereas your existing project is left unassigned.\nLeaving the option selected - you can select which branch will be your target of import:\nUse Collaboration - the \u0026ldquo;Collaboration branch\u0026rdquo; will be used as a target (\u0026ldquo;master\u0026rdquo; in this case) Create new - you might create new branch Use Existing - select required branch you want use as a target In my case - I will create new branch \u0026ldquo;InitAdfCode\u0026rdquo;:\nRepository Settings - all complete Once you click button \u0026ldquo;Save\u0026rdquo; - new branch is creating and one commit containing all resources (as JSON files) will be made. The following picture shows newly created branch and structure of folders.\nNewly-created branch Structure of folders. Familiar? Certainly. The latest step is to pick up working branch. As I have just created the new one - I want to us it:\nOnce confirm that - you will see that UI is connected to the selected branch in GIT repository:\nAt any time you can change working branch selecting required from the list.\nMaking changes Whenever you change any object in ADF v2 - Save button is become enabled. Once you click SAVE - all changes have made so far will be committed (PUSH) to the related folder in GIT repository. The only exception from that rule is when delete objects - the change is pushing automatically to the repo:\nAzure DevOps - history of changes pushes by ADF v2 Publish We are able to publish an edited ADF which generate publish branch which contains ARM templates that can be used for deployment purposes. Beforehand you must merge your changes from working branch to the master (or other branch pointed as collaboration branch).\nTo achieve that - use action Create pull request [Alt+P] (top-left corner) which does nothing but redirect you (in new browser\u0026rsquo;s tab) to project in Azure DevOps on a site with New Pull Request. Default direction is a \u0026ldquo;collaboration branch\u0026rdquo; (i.e. master) which can be the only source you can publish from. I\u0026rsquo;m not going to cover that topic in this post at all.\nSwitch assign repo to a different one You have might noticed that you can have got only one assigned GIT repository. Which is completely fine (you can juggle within branches). But what if you make a mistake or just change your mind and need to re-assign from one GIT repo to another one? Still you can do that, but in that case having only one place where that action is available: Overview dashboard. Basically, the operation is based on remove Git and assign it again, so firstly you must do:\nDetach / remove Git (repo) Removing Git repo from Azure Data Factory is possible only in one place: Overview dashboard. In top-right corner you will see (or not. I haven\u0026rsquo;t for first time) small Git icon:\nClicking the icon you will gain access to information about attached GIT repository. At the bottom you\u0026rsquo;ll find \u0026ldquo;Remove Git\u0026rdquo; button. Once finish that - you can repeat the assigning process from the beginning.\nThanks for reading!\n","date":"2018-10-05T00:31:36Z","image":"/uploads/2018/10/213449-ADFv2-Overview-dashboard.png","permalink":"/2018/10/setting-up-code-repository-for-azure-data-factory-v2/","title":"Setting up Code Repository for Azure Data Factory v2"},{"content":"Entire last week was dimmed by MsIgnite, hence take a look at videos from that conference:\nMicrosoft Ignite 2018\nSome sessions recorded and available over there.\nCosmosDB at Ignite 2018\nWhole playlist of sessions about Microsoft Cosmos DB.\nOne of the well-hidden videos.\nhttps://www.youtube.com/watch?v=iCuI0n9dA7w\nhttps://www.youtube.com/watch?v=F4GjeHct1O0\nAzure Data Factory: Visual Data Flows for Data Transformation (Preview)\nRead more about a new feature in ADF - Data Flows.\nUsing Power Query And Microsoft Flow To Automate The Creation Of CSV Files\nPower Query is available in Microsoft Flow - have a look what trick you can do.\n","date":"2018-10-01T09:17:08Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/10/last-week-reading-2018-09-30/","title":"Last week reading (2018-09-30)"},{"content":" Introduction Itzik Ben-Gan is a Mentor and Co-Founder of SolidQ. A Microsoft Data Platform MVP (Most Valuable Professional) since 1999, Itzik has delivered numerous training events around the world focused on T-SQL Querying, Query Tuning and Programming. Itzik is the author of several books including Microsoft SQL Server High-Performance T-SQL Using Window Functions, T-SQL Fundamentals and T-SQL Querying. He has written articles for SQL Server Pro, SolidQ Journal and MSDN. Itzik\u0026rsquo;s speaking activities include SQLPASS, SQLBits, SQLTeach and various user groups around the world. Itzik is the author of SolidQ\u0026rsquo;s Advanced T-SQL Querying, Programming and Tuning and T-SQL Fundamentals courses along with being a primary resource within the company for their T-SQL related activities.\nThis talk has taken place during SQLDay conference in Wroclaw (Poland), on 16th May 2018 (Wednesday).\nInterviewers: Kamil Nowinski \u0026amp; Damian Widera.\nDo you know what Itzik does like doing the most? What kind of thing is important when writing a book? Why worth to be at conferences and also when he has started working with Microsoft SQL Server databases? Find out more in this talk.\nTranscript Damian Widera: We did already the introduction so you know what it will be all about. I mean, in a nutshell. Would you like to introduce yourself to our listeners? So, who are you? Where are you from? What are you doing?\nItzik Ben-Gan: Sure. So. Well, first of all, I have a hard time answering where I\u0026rsquo;m from because I was born in Georgia. When I was one year old, my parents took me to Israel. And then about eight years ago my wife and I and our two dogs we moved to the US.\nDW: So you are very international right now.\nIBG: So if the question is where do you live? It\u0026rsquo;s in Redmond Washington in the Seattle area. And my name is Itzik Ben-Gan but I guess you said it\u0026rsquo;s in the introduction.\nDW: Yes.\nIBG: I\u0026rsquo;m with SolidQ, I\u0026rsquo;m one of the founders of the company. We started back in 2002, at the time we were six people. And the thinking was first of all people who kept meeting each other in conferences and events like this and like each other and like working with each other, and we all like SQL. And we thought that together we could join forces and try to cover more than each of us could do alone. It worked quite well. So, I\u0026rsquo;m still part of the company and my main focus is T-SQL. I did a lot of things related to databases. I started with them around the early 90s, 91 or so. But at some point, I started focusing more on T-SQL and Query Tuning. So pretty much anything that has to do with the performance of the code. So internal structures, query plans, writing efficient code. It\u0026rsquo;s not just about the range of the physical structure. You often have different ways to write the code that under different conditions each will tend to work better. So that\u0026rsquo;s pretty much what I focus on, and I still like it very much. I think I\u0026rsquo;m fortunate in the way that I can do what I like and not everyone can do this. So mostly teaching. So it\u0026rsquo;s T-SQL and mostly teaching, that\u0026rsquo;s very much what I focus on.\nDW: And do you like teaching, spending time with people? Because teaching can be pretty exhaustive because you have to spend like two, three, sometimes five days with a group, depending on how long the training is.\nIBG: Yes, so teaching is what I really love doing. So all the rest I do just to actually enable the teaching. And I know that for different people there\u0026rsquo;s different things that they like doing that make them happy. For me that\u0026rsquo;s this part. Actually for me it\u0026rsquo;s much easier than anything else. I talk about what I like, I have people that like listening to it, which is also nice.\nKamil Nowinski: And then it\u0026rsquo;s much easier to do that.\nIBG: And it\u0026rsquo;s also the area that I like focusing on. So yes, I really love it.\nDW: And how many books have you written because I have a lot of them. And we still have a bookstore in the venue and there are a lot of your books about T-SQL, windowing functions. The windowing functions is really my favourite one. It\u0026rsquo;s not that big but it\u0026rsquo;s very condensed.\nKN: And from my perspective, you\u0026rsquo;re also focusing on this windows functions. Very often I\u0026rsquo;ve seen that you talk about that subject.\nIBG: Yes. First of all the number, I believe it\u0026rsquo;s around 10-11-12, something in that area. The window functions specifically is also my favourite I have to say. And the reason that topic is so interesting and that I focus on it so much is that the original intention behind getting it was kind of a specific type of calculation which is data analysis. Doing things like aggregates and so on. But in a more, I would say sophisticated way. Just a very different way than using group queries using subqueries, those kinds of things. It has some kind of ingenious ideas about it. You group data, the grouping causes you to lose the detail. And often you need to mix, when we do calculations like percent of the current value out of a group total and stuff like this. You don\u0026rsquo;t want to lose the detail information. And with subqueries, you get an independent view of the data, whereas with windowing you apply the logic after all of the query elements have already been processed, not an independent view of the data. And that\u0026rsquo;s usually what you want. So it\u0026rsquo;s a more sophisticated tool but I think what\u0026rsquo;s interesting about it is that you can use it not just for the original types of problems it was designed to solve. That with time that you work with it more, you discover you can do much more generalized things. And it\u0026rsquo;s as if nowadays you have these two kinds of problems in SQL. Not problems, I would say for almost any kind of problem I should say. There are two kinds of solutions. As if you have this fork: one is the classic predicate-based solutions that use subqueries, that use JOINs with predicates and the ON clause / WHERE clause. But often it\u0026rsquo;s more complicated for people to think in those terms and it turns out to be far more expensive in many cases in terms of how the SQL Server engine ends up optimizing. And you can find all kinds of creative ways to use window functions to solve those problems way more efficiently. So areas you would not expect to see them, you know temporal data. When you need to deal with intervals, doing techniques like pecking intervals, finding the maximum number of concurrent intervals. You wouldn\u0026rsquo;t think initially there\u0026rsquo;s anything window functions have to do with it. So nowadays, I think of problems typically as: do we solve it with a predicate approach or do a window function approach. So it seems to people like a narrow area is much more general than people realize before they start using them. And that specific book, I wrote it back for 2012, when the major editions like the window frame was added or the leg and lead and statistical functions. They make it much more powerful and I\u0026rsquo;m actually waiting for the next SQL Server version, what they call vNext now, and we don\u0026rsquo;t know when it will be released to update. That\u0026rsquo;s the next book I want to update. But it is a very specific feature I\u0026rsquo;m waiting for in order to update the book.\nKN: I wanted to ask if Microsoft is extending those functions?\nIBG: So that they didn\u0026rsquo;t extend the functionality. So since 2012 they didn\u0026rsquo;t add new kinds, even though they are still a whole bunch of very, very interesting features in the SQL standard and that you do see in some other platforms that were not yet implemented in SQL Server. But the 2016 version is when they added this new optimisation of window function using this batch mode, window aggregate operator.\nKN: Exactly. It\u0026rsquo;s a huge difference.\nIBG: It\u0026rsquo;s an unbelievable difference in performance for the cases that satisfy the requirements for using it. But anyway, it\u0026rsquo;s amazingly fast. It\u0026rsquo;s just that it uses a batch mode kind of processing, where instead of processing one row at a time, it will process one chunk of rows at a time, a chunk of 900 rows. But currently, there is a condition in the engine that the batch processing can only be used if this columnstore index is present. Even if it could be very relevant if the data is pulled from row store index, and in fact if you\u0026rsquo;re able to create a supporting index, with a B-tree-based row store index you could avoid the sorting that is necessary to use the batch mode. Sorry, if you use columnstore, that will then enable a batch mode.\nKN: A very expensive operation.\nIBG: Yeah. So you have a regular columnstore index, you either need to pull it more compressed but then sort it and then have the window calculation done in batch mode or pull it from row store, already presorted but then use the row mode. Well, apparently if you create some kind of a columnstore index, even if it\u0026rsquo;s not used, the optimizer can already enable the batch processing despite the fact that it\u0026rsquo;s actually pulling the data from row store, as long as a columnstore index is present.\nDW: It\u0026rsquo;s just like a trick or workaround.\nIBG: Yes it\u0026rsquo;s like a trick but you can get to so many people and explain to them: look, now you can actually get it, but you have to create this index even if it\u0026rsquo;s a meaningless filtered index. But you need to do it and many people will not hear about this or not know about this. Obviously it\u0026rsquo;s better if in the engine they remove the connection between the presence of columnstore and the use of batch mode. And in the last press conference, Microsoft announced already a number of\u0026hellip; they publicly announced a few things that they are planning, what they call vNext and one of the things they were talking about was the removal of the dependency. So for all those systems that will not need to use any tricks or create any special indexes, it will be kind of available out of the box. And I think it will be a huge performance difference, without doing any code changes or creation of any objects.\nDW: We don\u0026rsquo;t change the code but they have to change. And that might be complicated, I believe. It might not be a simple disconnection in code but\u0026hellip;\nIBG: Well, I think the reason they didn\u0026rsquo;t make it available already, they knew obviously that ultimately the plan is to have batch processing not tied only to columnstore. They obviously had this in mind. Because inherently it\u0026rsquo;s not supposed to be this strong connection. Yeah, the arrangement of columnstore data lends itself to building batches but no one says that pulling it from row store building batches and getting all the benefits shouldn\u0026rsquo;t really be an option. So I\u0026rsquo;m certain they had in mind this scenario for the long term but they wanted to do regression tests and make sure that there are no all kinds of cases where the caustic formulas are such that when it shouldn\u0026rsquo;t really use the batch processing, it does use it. So I\u0026rsquo;m guessing they wanted a much more stable environment. And currently when we use this trick, it could be that there are here and there cases where it didn\u0026rsquo;t make sense for the optimizing to use it but it uses it, I don\u0026rsquo;t know. But yeah, I\u0026rsquo;m very much looking forward to the next version because I think it will be kind of a big performance boost with no need to create objects and no need to change the code for those queries using window functions and many others. I mean it\u0026rsquo;s just a basic \u0026lsquo;group by an aggregate\u0026rsquo;, where currently most of your workload doesn\u0026rsquo;t really benefit from columnstore. But now those kinds of queries that do, just basic aggregates that do sorting, they do a lot of kinds of activities that just run much, much faster in batch mode. And many other improvements that they make. Almost anything that they do in the optimizer, they invest mainly in batch processing. And either no investment at all in the row mode or they do very little. So if you want to enable those new features, you need the removal of this connection. You probably heard of the adaptive query processing.\nDW: Yes, of course.\nIBG: It is another good example that if you take that the adaptive join, where you have a situation where in some conditions the optimizer needs to use a loop join, typically when you have a small number of rows involved in another small table. If you have a larger number, it needs to use a hash join. But then you have those parameter sensitive queries where you use a parameter in the store procedure and the first execution determines the kind of strategy it chooses and caches the plan and now we\u0026rsquo;re stuck with this one. Even if there are many executions with values that would need the other plan.\nDW: And it seems to be a typical problem still.\nKN: Yeah, a very popular problem with parameter sniffing.\nIBG: What people end up doing is they end up actually using a recompile option to force the system to create a new plan every time. And this very smart group of operators did it in 2017, called adaptive query processing. One of them is this kind of smart join where the outer branch is identical for both loop and hash but there is one inner branch for the hash and another inner branch for the loop and then it compiles this threshold. And if the number of rows involved is below the threshold, it activates the hash branch, actually it\u0026rsquo;s the loop branch, below, and then on or above it activates the hash branch. And then you don\u0026rsquo;t have to force a recompile so it is able to cache the plan with this more dynamic operator and activate the right branch during execution. The tricky part is what does it need in order to activate it? Batch processing. This is an operator that can only walk on the batch more so if you didn\u0026rsquo;t have the columnstore index, it\u0026rsquo;s not going to consider it.\nKN: Yes.\nIBG: So you realize that this can only make sense obviously to remove the connection.\nDW: But it looks like maybe for people who just write a query, it\u0026rsquo;s like \u0026ldquo;oh, it\u0026rsquo;s a simple query\u0026rdquo; but as we discussed, they might not know how complicated the query optimizer is and how many decisions it has to take every single query to optimize even a very simple statement.\nKN: Yeah, at the beginning it looks like a very simple process but internally it\u0026rsquo;s very long let\u0026rsquo;s say and very complicated.\nDW: And you can still be fast. Good enough, not the best one, but very, very fast so the compilation is no longer than the query actually. It could be looking for the very, very best plan forever.\nIBG: This is interesting the point you mentioned, because they could a lot more rules to the optimizer that don\u0026rsquo;t require too sophisticated additions. It\u0026rsquo;s just that they would make the optimization process even longer and counterproductive. So a good example where people often fall into this kind of trap is with the search arguments. Where if you do some kind of manipulation, like there\u0026rsquo;s this common one where the input for a procedure is a parameter but where the values can be either non-nulls or sometimes the input could be null, like give me the orders with the ship date is the specific one but all orders were not shipped yet will have a null ship date. And you want to support a null prompter. So in order for the filter to work, people often apply calculation like isnull on the parameter and the column, and replace with something else that cannot appear in the data, but the same in both sides, and then you get the correct treatment but you lose the searchability of the filter, because you manipulate the column. So this example or when you say \u0026ldquo;give me the rows where the year of the date column is equal to some year\u0026rdquo;, you lose searchability and then it does a full scan. And people often don\u0026rsquo;t realize this and then when you explain to them, \u0026ldquo;well you know, if you change it to not doing manipulation but with the same meaning, like column equals parameter or column is null and parameter is null, it\u0026rsquo;s a searchable kind of a filter and it can use the index. And then they say \u0026ldquo;well, couldn\u0026rsquo;t Microsoft add this to the optimizer and not ask to do the rewrite?\u0026rdquo; And they don\u0026rsquo;t always realize that yes, they could but then if you overwhelm the optimizer with too many rules, the optimization process like you said could become longer and even longer than it takes for the query to run.\nKN: Longer and more generic and that could lead to much more problems.\nIBG: And Microsoft I\u0026rsquo;m sure intentionally didn\u0026rsquo;t put it in the optimizer. Rather they said: \u0026ldquo;if we can give it as a best practice to people then we can pull all this logic out of the optimizer and then people will just follow: here is the pattern that is recommended to use when you write the filter\u0026rdquo;. That is true. But what I think is interesting in this kind of whole world is that both the optimization part is very kind of deep and there are all kinds of levels you could go into. You have people like Paul White, who knows the optimizer to such levels that very few people in the world manage to reach. But then there are many other people that do get into some level but decide at this point we stop. So there are different levels and the same I can say about the SQL itself. That you start working with it, you get kind of a false impression that it\u0026rsquo;s an easy, simple kind of language, probably a week into the process of learning SQL you already have queries with joins and where and group by, and you feel very confident, and it\u0026rsquo;s after 10 years that you start realizing\u0026hellip; well, maybe I\u0026rsquo;m exaggerating with 10 years but the more you work with it, you realize there\u0026rsquo;s a lot of depth there and you can dedicate your life to this topic.\nDW: The more queries you write, the more you\u0026rsquo;ll see that it\u0026rsquo;s not that easy. And when I start training with people in T-SQL, they are always like: \u0026ldquo;oh, it will be simple. What are we going to do for the next three days? It\u0026rsquo;s just a select\u0026rdquo;. And then we go deeper and deeper and deeper, and it\u0026rsquo;s not a very deep level and they see, \u0026ldquo;oh it\u0026rsquo;s not that easy\u0026rdquo;.\nIBG: And more questions after three days, I guess.\nKN: So we are going in the direction that the world generally works with more and more data, that\u0026rsquo;s why we have columnstore indexes, that\u0026rsquo;s why we should at the beginning focus from the more analytical point of view when we are looking at the data. So do you think that in the nearest future we\u0026rsquo;ll be talking more about the columnstore index because it\u0026rsquo;s connected like we said?\nIBG: Well, first of all for sure columnstore index will play a big role in analytics, batch processing, specifically the optimization part, not just with columnstore, but this will be I think just a small part. But what you\u0026rsquo;re completely right is that it\u0026rsquo;s more and more data, it\u0026rsquo;s obvious to everyone. And by the way there is this author called Yuval Noah Harari, he\u0026rsquo;s a historian, teaches history at a university and he wrote two books, one about Sapiens and the other one he calls Homo Deos. So the first is just the way homo sapiens kind of evolved. But the second one, he talks about where he sees the future going, very kind of original type of thinking. So strongly recommended, but he talks about data as becoming kind of\u0026hellip; I know for some people it would sound a bit painful but it kind of a religion in a way. That everything is around data more and more, more of our decisions are around data. And take data together with AI algorithms and so on, at some point those algorithms will know humans better than we know ourselves, because they can capture a lot more data and process it than we can. Pretty fascinating but yes, certainly that\u0026rsquo;s what you see, it\u0026rsquo;s more and more critical part of decision making. So we are in the right place.\nDW: Somewhere in the middle maybe, when we see a lot of data but not that much of data. But if you think that maybe right now\u0026hellip; I\u0026rsquo;ve seen a presentation about big data in Twitter, in a minute there are like 600 millions of tweets every minute. This is impressive. Every minute, every day.\nKN: If you want to follow some special tags, hashtags and if you put something like machine learning but especially big data, it\u0026rsquo;s a lot of messages every single second. We can see a lot of your activities not only in the USA but also across the world. You\u0026rsquo;re travelling a lot. I remember that I met you\u0026hellip;\nIBG: I think it was in London.\nKN: I was so surprised that I can see you on that kind of meeting, that kind of local meeting. But then we had an occasion to chat to each other, very quick chat. So I know that you have some customers over there as well, because it\u0026rsquo;s a big city, so could you tell us how often you travel to Europe or maybe to other parts of the world?\nIBG: I moved to the US eight years ago. The main reason was because my work was mainly in the US, but ironically as soon as I moved, I started getting a lot more work in Europe. But like I told you, I grew up in Israel. Now my base is in the US, I travel a lot. I think something like 70-80% of my time I\u0026rsquo;m travelling. Probably half of it is Europe and half of it is the US. But yeah. So for instance next week, I\u0026rsquo;m in Sweden but I have a flight home on Thursday, I will spend two days at home, I will be back in Sweden. Then I will have a week in the US and then I will have a week in Nurnberg in Germany, it\u0026rsquo;s like this. I don\u0026rsquo;t like the flying part, but if you told me you have to do something else that you like less but you will stay always home, I\u0026rsquo;d prefer the way it is now. So at least I know that I\u0026rsquo;m doing what I like doing.\nDW: So when do you find time to write another book?\nIBG: Well the writing. First of all the writing is not just the point of physically writing. You do it in your head all the time. So I work always with new ideas related to SQL, related to optimization and I process them more in my head. The writing part is more like kind of opening a drawer and pulling one of those things and maybe more polishing it. The writing part does it because you force yourself to make sure that you have all the angles kind of covered and so on. So there is an important part of the writing process but I usually do it every two hours a day. So usually before the working day or after the working day, I will have something like half an hour in the morning and maybe an hour and a half or two hours in the evening, and it sounds like a little bit but every day you do it for half a year and there\u0026rsquo;s a book. That\u0026rsquo;s my process.\nKN: That\u0026rsquo;s interesting that it works for you because for most people it\u0026rsquo;s hard to start something new or focus on something new. It takes time, at least a couple of months for example.\nIBG: For me, it\u0026rsquo;s not a new thing, that\u0026rsquo;s the thing. There are things I\u0026rsquo;m working on all the time but it\u0026rsquo;s just a matter of putting them into words.\nKN: Maybe it\u0026rsquo;s not new. I mean different activities that during the day you just used previously.\nIBG: I don\u0026rsquo;t take vacation because I would normally take vacation if I wanted to just sit and write and do nothing else.\nDW: That\u0026rsquo;s what we do when we wrote one of our books. We had two weeks and during these two weeks, we just wrote it and it was a very intense moment.\nIBG: Yes, for some people I imagine that\u0026rsquo;s a process, for me it\u0026rsquo;s every day a little bit.\nDW: But we were in a group, there were four people and we had to do it then, because it was very hard to synchronize all four people during the entire year, so we spent two weeks just writing a book. That\u0026rsquo;s how it worked.\nKN: The next question must be about your work-life balance. Because I\u0026rsquo;m interested in what your approach to work-life balance is, how you can manage that.\nIBG: My wife and I are very good friends and we don\u0026rsquo;t have kids, we have two dogs. So obviously it\u0026rsquo;s easier.\nKN: So we already have the answer to our next question.\nIBG: But yes, we are very, very good friends and the fact that nowadays you have Skype and things like this, it makes it much easier. She used to travel with me years ago. It was fantastic but now because our dogs are very, very old, there are 15 years old, it\u0026rsquo;s impossible to take them in a car or close to impossible to put them in a can. So lately she stays with them, but every day we have Skype, we have WhatsApp and stuff like this. I sent messages to my wife this morning. There are no special kind of challenges other than the fact that I\u0026rsquo;m away. And I would like to be with her and with the dogs but we understand that this is part of life. That\u0026rsquo;s why for instance if I have two days between two projects in Europe, I will go back home and say \u0026ldquo;OK, I don\u0026rsquo;t want to suffer the jet lag and so on\u0026rdquo; and then I will just stay in Europe.\nDW: It would be easier just to move from here to Sweden as you said, because it\u0026rsquo;s like a two-hour flight from Wrocław.\nIBG: Yeah, but you see, if I was a lot at home, then of course, it would have made sense. But since I\u0026rsquo;m little time at home, then I will go back, spend a couple of days and go back. Maybe I will not change my watch. I will keep my watch on Europe time and be in the US but stay in Europe time. And then I would not need to go through a jetlag, we\u0026rsquo;ll see.\nKN: Maybe that will be the trick. So if you could describe your general activities, what would it be? I mean we already know that there is a lot of training but I think you\u0026rsquo;re writing some white papers for Microsoft?\nIBG: Yeah, there were a few papers I wrote in the past but I maintain a column for close to 20 years now and recently, by the way, I switched the column to sqlperformance.com.\nDW: Like last week you posted I think the last article, second part.\nIBG: Yes. I just started working with them a couple of months ago. So just my monthly column is now going to be in the SentryOne website. I really like both the company and the group, so being part of people like Aaron Bertrand (T), Paul White (T) and so on who also regularly write there is really amazing. But yeah, I write a monthly column. And by the way, for many people who kind of look for ways to enhance their knowledge and so on, sometimes you pick a topic that you\u0026rsquo;re working on and you\u0026rsquo;re writing on it. But sometimes when you know that you have every month a certain column that you need to deliver, you keep processing in your mind all kinds of ideas, like even if it could be a customer, could be a student that presents some kind of question or an idea and in your mind you already know you need to research this further and it becomes an article. So during this process you get to learn a lot. So for people who look for ways to expand their kind of knowledge and keep always fresh, maintaining a monthly column is a fantastic way. So you just don\u0026rsquo;t want to burn yourself so you don\u0026rsquo;t want to try and write two blog entries every week. And then OK, you do it for maybe a couple of months and then it\u0026rsquo;s just impossible for you to continue to produce good quality and deep enough topics.\nKN: Let\u0026rsquo;s start from something frequent but like one month\u0026hellip;\nIBG: I think it\u0026rsquo;s better to just ahead to decide on something more realistic that you can keep in the long run. So I\u0026rsquo;m running it since ninety-nine this column. Looking back, what are the things that for me contributed to the knowledge, one is the column, another is participating in forums, all kinds of back then was newsgroups and so on, where you are you have exposure to lots of different questions on one hand from different kinds of areas and also other people\u0026rsquo;s ideas. So there is a thread someone asked how do I handle this situation and then you first of all want to work on the task yourself without looking at other people\u0026rsquo;s solutions but then you come up with one idea, then you look at other people\u0026rsquo;s ideas and you get exposed to a lot of information and techniques and ideas.\nKN: So in which forum can we find you?\nIBG: Nowadays, actually I do less forums. I\u0026rsquo;m saying that in the beginning how I started getting a lot of ideas, it used to be newsgroups and NNTP newsgroups if you remember them, they were so nice. Currently there are MSDN forums, Stack Overflow\u0026hellip; Nowadays I do less and I really wish I did more of this even now. But thinking back where I gained a lot of the knowledge it was those things. The column I still maintain of course, like I said every month.\nKN: So apart of white papers, books and trainings, do you work for some specific customers, specific projects?\nIBG: First of all as a company, I said I\u0026rsquo;m with SolidQ. The company does a lot more customer work than the training part, is more my personal favorite things. So as much as I can control it, when they ask me what I want to do, I tell them I prefer much more training. As a company, a lot more customer-facing, and field work, some small projects like there\u0026rsquo;s a query tuning problem or some performance problem, and some very big projects, designing the whole systems and so on. But yeah, we have customers all over the globe. And nowadays it\u0026rsquo;s a big company with about 200 people. We have some regular customers, some new customers that are coming all the time. So it\u0026rsquo;s in all fields. I don\u0026rsquo;t want to name just one, but finances, health, government-related, pretty much everywhere.\nDW: And what do you like to do apart from T-SQL and working? Like hiking, swimming?\nIBG: I love hiking. I love friends, so this is one of the reasons why it\u0026rsquo;s so nice to have those conferences, even though I don\u0026rsquo;t do many conferences. But when people walk in a specific place, they meet their colleagues every day. For me I\u0026rsquo;m travelling all the time, I mainly see the students but it\u0026rsquo;s every week a different group. By those conferences I can meet people I know and catch up and so on. But hiking -- I love hiking so every year we usually have this kind of long hike, usually in the summer, something like a whole week in the backcountry. And it\u0026rsquo;s a group somehow that got connected through SQL. You probably know Lubor Kollar, he\u0026rsquo;s now retired but he was responsible for many very big, important features in SQL Server like partitioning, etc.\nDW: I was fortunate to meet him about 10 years ago. And I think it was a PASS conference in Dusseldorf. It was one of his last appearances. Now I see on Facebook he\u0026rsquo;s retired for some years. He\u0026rsquo;s with his kids but he\u0026rsquo;s still very, very active, many peaks taken from the mountains.\nIBG: So he started the idea. I think it was 2006. He started this idea of \u0026ldquo;let\u0026rsquo;s go do some interesting long hike\u0026rdquo; and then we did a segment of what\u0026rsquo;s called the Pacific Crest Trail. This is a very long trail that starts in Mexico and goes to Canada. We did a small piece of it. It was something like 70 miles I think, something like this. But since then every year we have this group that does that hike sometime in Europe, sometimes in the US. So one year we did\u0026hellip; in Ireland there is a beautiful hike called the Carry Way. It has this kind of shape like 8, but 110 kilometers. And so I love this and we call it SQL Hike because it\u0026rsquo;s mostly\u0026hellip; I mean, here and there we have people not from the SQL but it just happened because of our work. We meet each other, it\u0026rsquo;s how we make our friends. Yeah, Boris Baryshnikov (L), maybe you know him. He also did a number of important things in SQL like Resource Governor, he was responsible for the feature and so on. Dejan Sarka (T), Herbert Albert (L)\u0026hellip;\nKN: Do you need to specially prepare yourself for the speeches for example, if you\u0026rsquo;re speaking at some conferences?\nIBG: Usually you just want to have the first couple of sentences, you want them ready. But I don\u0026rsquo;t do this a lot of time ahead. So like before we start, maybe a couple of minutes before the session is to start, you want to make sure that you have like a couple of sentences ready. But then nothing special, it kind of flows. And for me, because I\u0026rsquo;m much more interested in the code part than slides and those things, and I do this regularly all the time, it\u0026rsquo;s kind of a very natural process. It\u0026rsquo;s easier for me, for instance, to start showing queries than to speak in English.\nDW: So the query language is easier.\nIBG: The query language is much easier for me. Because my brain is thinking in such patterns. The English part is still very hard for me because it\u0026rsquo;s not my native language but the SQL part is because my brain is all day all night thinking in SQL terms, it\u0026rsquo;s kind of natural. And also, for people I think it\u0026rsquo;s much better when they see the code developing as opposed to if you just show a piece of code on a slide or just highlight a piece of code in a script and run it, because then it\u0026rsquo;s very hard for them to see how it evolved. So if I write the queries and explain the thinking during the writing part, then I think it gets absorbed much better. And also, when they will work on the technique themselves, I hope they will think in such terms. Start thinking about the FROM clause, what table is involved, what do you join, and so on.\nDW: So how did you start with conferences, when was your first conference?\nIBG: First conference I think it was PASS, maybe 2000, maybe 2002. This was like the first big conference. But there were before, through the MVP, what used to be called SQL Server MVPs and now is the Microsoft Data Platform MVPs, there were Summits, and we still have those, with Microsoft. But those are internal conferences, where Microsoft mainly provides information about future plans and stuff like this. But as a speaker, I think the beginning was maybe 2002.\nKN: Do you remember your first conference?\nIBG: It was PASS Summit. I will never forget it, that\u0026rsquo;s for sure because I had a very big stage fright at the beginning. The first conference I was supposed to give a lecture, I didn\u0026rsquo;t know originally, but when I checked the agenda and looked at the organization of the rooms, my lecture was supposed to be in what they call the ballroom 6C. And compared to all the other rooms that looked like small squares in the picture, this took almost the whole page and I realized there\u0026rsquo;s going to be something like 500 people there. And you know that the fear of public speaking is a big fear for many people and for me it was like this. And I came from Israel, so I was first jetlagged. And with the excitement around I\u0026rsquo;m going to speak to so many people. I didn\u0026rsquo;t sleep for three days and I tried all kinds of things, I just literally didn\u0026rsquo;t sleep for three days. And then when I was supposed to give the lecture, I just went to the hotel I asked for a bucket, and I went to fill it with ice and I filled the sink with ice and water and I put my head inside. Then I took a quadruple coffee from Starbucks and I started the lecture. It went quite well, amazingly but\u0026hellip;\nKN: At least for the one hour that you needed.\nIBG: I was not concerned at all about the topic or being able to show interesting stuff or the demos will not work. My concern was more to be able to start to speak, to be able to breath and not get excited. But since then, it became easier and easier and I still get very excited when I speak, but it’s now kind of a healthy type where it keeps your adrenaline up.\nKN: So do you still feel some stress?\nIBG: Yes, certainly, I still feel, but I\u0026rsquo;m saying, I feel that it\u0026rsquo;s now a healthy kind of stress. If you have less of it and you start becoming it different or something, you would have a different type of lecture and the quality would be lower. So I think now it\u0026rsquo;s at a good level. But it\u0026rsquo;s 20 years after.\nDW: So now you don\u0026rsquo;t need to bring the ice and put into the sink.\nIBG: Exactly, now I don\u0026rsquo;t need ice. Still coffee\u0026hellip;\nDW: But the ice part is no longer needed.\nIBG: It was funny, though.\nKN: With which version of SQL Server did you start working?\nIBG: I started with 6 5. This was my first.\nDW: So it\u0026rsquo;s 90s?\nIBG: This was the very early 90s. Wait a minute, early 90s it was still other database platforms. So it\u0026rsquo;s more mid-90s towards the late 90s with the 6.5, you\u0026rsquo;re right. So back then I was working in this big beer factory and the database was a DB2 database and AS/400. And also I pulled off data from AS/400 into an access database and did a lot of financial analysis, costing, sales analysis, those kinds of things. And by the way, back then I used to do a lot more kind of all-up work. And I used the original product that Microsoft then acquired to create what nowadays is analysis services in SQL. So back then it\u0026rsquo;s a company called Panorama, an Israeli company that developed this cube engine and also the viewer that they had, and in that company, we used their products. So I created cubes. My first cube was beer. Beer-related cube. Then Microsoft did an acquisition of the engine, so that company kept developing the viewer. They still own the viewer but the engine became first OLAP Services and then later they made it Analysis Services.\nKN: That\u0026rsquo;s an interesting part.\nIBG: Yeah. And the funny thing is when I left the job, there was a drink party where you say goodbye to everyone, bring your presents and so and, and then my boss told me \u0026ldquo;but I need another cube\u0026rdquo;. Just before I left, after we already drank and said hello. \u0026ldquo;But I really need this one more cube\u0026rdquo;. So I sat down and had to create another cube before I left.\nKN: Interesting.\nDW: I have one more question about\u0026hellip; Do you have any advice for people who would like to start, I don\u0026rsquo;t know, at the conferences and they just fear about how to start talking, how I should prepare?\nIBG: It\u0026rsquo;s hard to say because I guess it\u0026rsquo;s a bit different for different people, but what I would say is that it\u0026rsquo;s an important part if you want to get the exposure\u0026hellip; For me, for instance, I prefer the classroom setup than the conference but in order to get people in my classes, the way I create the exposure is the public speaking and writing and so on. And they\u0026rsquo;re all healthy parts of the whole kind of situation that you have. So I would say just get over your fear, pick a topic that you are very comfortable with, and just do it. Maybe to try and do things like what I mentioned, to try and at least tell yourself the first couple of sentences before you start. Picking a topic that you really like because after you get over the first couple of minutes of excitement, then it becomes more natural. So do it. I would say if you\u0026rsquo;re planning to become part of this community and you want to be able to, especially if you\u0026rsquo;re independent. If you\u0026rsquo;re working in a certain job, you don\u0026rsquo;t need the exposure so much but if you\u0026rsquo;re independent, you\u0026rsquo;re consulting or teaching and so on, you need to get your customers and that\u0026rsquo;s a great way to do it.\nDW: And how important is the community? Do you think it\u0026rsquo;s important?\nIBG: Yeah I think first of all the SQL community is a very lively community. And it\u0026rsquo;s a big part of your knowledge. So first of all you meet each other, you make friendships, you make connections. But then a lot of times when you sit and chat, some of it is just jokes but some of it is ideas. So you don\u0026rsquo;t always get your SQL ideas from a customer or from students. Sometimes it\u0026rsquo;s just the colleagues that discuss \u0026ldquo;hey, I work on this thing and I face this challenge\u0026rdquo;. Definitely, it\u0026rsquo;s a way to get to know other people, to get exposed to other ideas, and our reality is that work is such a central part of our lives for many of us, that we make our friends through this kind of environment. So I think the community\u0026hellip; both the community and you know, you look at the teams that develop the product and it\u0026rsquo;s a very open kind of group.\nKN: For new ideas?\nIBG: No, I\u0026rsquo;m saying that SQL Server developers, the development team, the Tiger team, the customer advisor team, they\u0026rsquo;re very, very open. They see the importance of the community and the interaction with them, they\u0026rsquo;re very accessible. We meet them at conferences, we chat with them. Obviously, they will not tell you every single thing they are planning because they simply can\u0026rsquo;t, but still, what they can\u0026rsquo;t talk about, they\u0026rsquo;re very open, very friendly. So I see it as a very positive kind of community and part of the world.\nKN: Currently I think it\u0026rsquo;s going to Azure basically and in the next couple, maybe several years there will be more and more systems and data pushing to the cloud. So how do you think it\u0026rsquo;s from your perspective or from the perspective of people who are working with performance, is it an important part? I mean, all those people lose their job or maybe lose the contract or customers?\nIBG: Not at all. The way I see it is just like nowadays, do you worry about the electricity that you\u0026rsquo;re using for your laptop or something like this? Historically, people used their own devices to create energy but at some point, it just doesn\u0026rsquo;t make any more sense. So you just plug into a socket in the wall and you don\u0026rsquo;t think too much about electricity. So in a similar way I guess the cloud will be where things will be running and where the data will be stored but then again you will connect to a database. And nowadays it\u0026rsquo;s already like this. If you connect with your SQL Management Studio, does it feel any different when you connect to a cloud or when you connect to the box product? Not really. And the way I write queries is the same, the way I tune them is the same, the indexing, the query plans\u0026hellip; You just don\u0026rsquo;t need to worry about patching, you don\u0026rsquo;t worry about many of the things that are related to the hardware itself\u0026hellip;\nDW: A lot more possibilities because you can connect, let\u0026rsquo;s say, the bricks together. You have a block like SQL, you have an analytics block, you have reporting blocks, whatever, machine learning blocks, and you can build very good solutions.\nIBG: Of course, the people that were very, very focused on hardware for instance, they have less relevance when the product is in the cloud. But for people who deal with the actual code\u0026hellip;\nKN: From this perspective it would be the same. But in the cloud, there are more automated things like performance, query store, and AI would be engaged in this, so probably there would be less work from the performance tuning.\nIBG: Well, from one aspect of performance tuning. So yes, the hardware tuning is not relevant anymore but index tuning for instance, doesn\u0026rsquo;t really change.\nDW: It\u0026rsquo;s a different environment so different patterns maybe and different things to look at.\nIBG: So for me at least, being focused on T-SQL and query tuning, I don\u0026rsquo;t see any difference. What I like actually about the cloud, even though many companies still don\u0026rsquo;t use the cloud. And maybe it will happen quickly or not, how they will transition, if they will transition. But for me, the interesting part about the cloud is that now Microsoft\u0026hellip; First of all the code base for SQL Server and SQL Database is the same codebase. It\u0026rsquo;s just that every so often they freeze a build and release it as a box product. But they get a lot more feedback compared to when there was no cloud involved. They have literally millions of databases, they get a lot of telemetry information. The feedback cycle, both in terms of seeing what is used more, what is useless and what are the problems, they manage to deal with those in a much more efficient way. And this results in much better box product because all of the experience there from the cloud, we end up with a better box product as well. And also you start seeing features being released much more frequently, and they usually appear first in the cloud and later in the box products. So even for the purposes of being able to play with a feature and to learn about it, it\u0026rsquo;s convenient to have access to the cloud. So I see it as a positive result, even for people who don\u0026rsquo;t use it. And I keep serving. Like when I teach, I check with people how many are using the cloud, and I get out of maybe 12 people, one is using the cloud. It could change. It very likely will change but I\u0026rsquo;m saying, even if the majority now don\u0026rsquo;t use it, they enjoy from it, whether they know it or realize it or not. They do gain benefits from it. And eventually, I see it as a service that you won\u0026rsquo;t need to think about too much. The problem I see nowadays is that there\u0026rsquo;s a lot of focus that the companies who provide the technology put on the cloud obviously because they want customers to move to the cloud. But the way I see it, it\u0026rsquo;s kind of inevitable. And the point where we will be completely there is when we will stop talking. It\u0026rsquo;s when we will be in the cloud but we won\u0026rsquo;t need to think about it. Just like we don\u0026rsquo;t keep talking about electricity today, even though we use it all the time.\nDW: It will be so natural. The cloud is just there.\nIBG: And you will just focus on the technology, not on the infrastructure behind it. That\u0026rsquo;s the day when we will really be there. As long as you keep hearing companies talking about the cloud, probably we\u0026rsquo;re not there yet, because they\u0026rsquo;re trying to get customers to move to the cloud. That\u0026rsquo;s why they are talking about it so much.\nDW: We are in transition somewhere at some point. Some companies still are on the premises but more companies are now trying to think about what and how we can benefit.\nKN: This process is ongoing. OK, Itzik. Thank you very much for accepting our invitation.\nIBG: Thank you for having me here at the conference, I\u0026rsquo;m really enjoying it.\nDW: Yes, people are really amazed about your sessions. My colleague, who I worked with at the university, told me that he\u0026rsquo;s literally hypnotized. And he had a positive push and kick to learn something new. He was not aware that this is so fascinating. That a query can be so fascinating. So it was really amazing. So people are really, really thankful for this workshop and the lecture. And they still have one left today.\nKN: At the end, because I forgot. Tell us where we can find you, in the Internet for example.\nIBG: So, first of all, like I said, I\u0026rsquo;m with SolidQ. So if you go to solidq.com you can see our company. And my personal stuff like the book related content and T-SQL-related resources that I maintain, I have a personal website that uses the same domain. Just starts with tsql.solidq.com instead of the www.solidq.com. And on Twitter @ItzikBenGan. Every time there\u0026rsquo;s a new article or a book, I will usually send a tweet.\nDW: And of course, SQLPerformance right now?\nIBG: SQLPerformance like I said, is the blog. So now it\u0026rsquo;s a monthly column basically. I don\u0026rsquo;t think of it as a blog. I started writing it before the term \u0026ldquo;blog\u0026rdquo; really existed. Nowadays you call them blogs. So it\u0026rsquo;s a monthly column. Every month there\u0026rsquo;s a new idea, usually related to advanced SQL or query tuning topics. And if anyone really wants deep knowledge about T-SQL, then I run regular 5-day classes, so they can just look up in the tsql.solidq.com website and in the courses section, there\u0026rsquo;s a schedule where I am in the upcoming weeks, and 5 days of just T-SQL and query tuning content.\nKN: They can join the class. Great, perfect.\nDW: OK, so thank you very much again.\nIBG: Thank you.\nUseful links: Itzik\u0026rsquo;s twitter: @ItzikBenGan\nItzik Ben-Gan on: SolidQ\nItzik\u0026rsquo;s posts: SQLPerformance \u0026amp; books\n","date":"2018-09-24T08:00:16Z","image":"/uploads/2018/09/ASF013_SQLDay_with_Itzik_Ben-Gan_1200.jpg","permalink":"/2018/09/asf-014-itzik-ben-gan-interview/","title":"ASF 014: Itzik Ben-Gan interview"},{"content":"Good day SQL folks!\nIGNITE IS COMMING!!!!\nStay tuned on our twitter this week but before you check in to the session on Ignite here you are - some exciting news from the the Data Platform world. You can join online if you are not in Orlando.\nLocks Taken During Indexed View Modifications\nBrent Ozar says he is always a little nervous doing indexing and here you can read why is that.\nAzure IoT Hub + Azure function + Azure Cosmos DB — Walkthrough\n*** Good for Monday morning *** This is a simple walkthrough about adding a serverless backend to your existing IoT system.\nDeveloper’s Choice: Hinting Query Execution model\nAn article describing a new hint available in the SQL Server 2016 SP1. You will be able to tweak the Query Optimizer for a specific intent, including the hints FORCE_LEGACY_CARDINALITY_ESTIMATION and FORCE_DEFAULT_CARDINALITY_ESTIMATION.\nStructured streaming with Azure Databricks from IotHub to Cosmos DB\nIn this article it is explained how to built a data ingestion architecture using Azure Databricks enabling us to stream data through Spark Structured Streaming, from IotHub to Comos DB.\nUnderstanding Neural Networks. From neuron to RNN, CNN, and Deep Learning\nNeural Networks is one of the most popular machine learning algorithms at present. It has been decisively proven over time that neural networks outperform other algorithms in accuracy and speed. With various variants like CNN (Convolutional Neural Networks), RNN (Recurrent Neural Networks), AutoEncoders, Deep Learning etc. neural networks are slowly becoming for data scientists or machine learning practitioners what linear regression was one for statisticians.\nCheers,\nDamian\n","date":"2018-09-24T06:00:20Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/09/last-week-reading-2018-09-23/","title":"Last week reading (2018-09-23)"},{"content":"- \u0026ldquo;my brain is all day all night thinking in SQL terms\u0026rdquo;.\nDo you know who has said that?\nWatch the video trailer and guess who is the next guest.\nThe next episode of #AskSQLFamily #podcast is almost ready. #StayTuned\n","date":"2018-09-21T23:04:48Z","image":"/uploads/2018/09/ASF014-video-invitation.jpg","permalink":"/2018/09/asf-014-video-invitation/","title":"ASF 014 video invitation"},{"content":"Azure Data Factory V2 is a great good tool for processing big data. It\u0026rsquo;s very scalable, flexible and modern cloud data integration service.\nYou can compose and manage reliable and secure data integration workflows at scale. Use native ADF data connectors to move and transform cloud and on-premises data that can be unstructured, semi-structured, and structured with Hadoop, Azure Data Lake, Spark, SQL Server, Cosmos DB and many other data platforms.\nVersion 2 of Azure Data Factory was released to general availability (GA) about three months ago. Comparing to previous ADF version - it\u0026rsquo;s completely new product, contains own UI improving developer\u0026rsquo;s experience, has many various connectors (including non-Microsoft systems) and capabilities to run Azure Databricks as one of the steps.\nNo wonder that completely refurbished product has a new module in PowerShell: AzureRM.DataFactoryV2.\nToday, I would like to show you how to use one cmdlet to deploy DataFactory v2 using PowerShell script which can be used in order to automate the deployment process afterwards (next post will be about it).\nQuite recently, I wanted to automate the deployment process using files being kept in a code repository.\nThis is important as the files are divided by type of object - as not as the same like ARM template files exported from Azure portal.\nAs I\u0026rsquo;m a big advocate of PowerShell for some time - it was obvious to use PowerShell in this case (not mentioning that it should be natural choice).\nThere are two separate modules: one for V1 and another one for V2.\nOnce checking documentation - I found that we must keep a specific order of deploying the pieces:\nCreate a data factory (that was obvious) Create linked services Create datasets Create a pipeline So, I have started preparing script to deploy ADFv2:\nSet-AzureRmDataFactoryV2 -Name \u0026#34;KamilExp-adf2\u0026#34; `\r-ResourceGroupName \u0026#34;rg-somethingbad\u0026#34; `\r-Location \u0026#34;westeurope\u0026#34; Then I have got the following error:\nSet-AzureRmDataFactoryV2 : HTTP Status Code: NotFound\rError Code: NotFound\rError Message: Operation returned an invalid status code \u0026#39;NotFound\u0026#39;\rRequest Id: 6b7eb2a5-b8dc-493c-b538-e3f14bcce730\rTimestamp (Utc):09/12/2018 16:35:14\rAt line:1 char:1\r+ Set-AzureRmDataFactoryV2 -Name \u0026#34;KamilExp-adf2\u0026#34; -ResourceGroupName \u0026#34;rg ...\r+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r+ CategoryInfo : CloseError: (:) [Set-AzureRmDataFactoryV2], ErrorResponseException\r+ FullyQualifiedErrorId : Microsoft.Azure.Commands.DataFactoryV2.SetAzureDataFactoryCommand Increasingly the description of errors says you\u0026hellip; nothing. I got used to.\nAfter an hour of investigation - I have found the issue. One fault was me (the original one), but the second one was caused by module (lack of information).\nHint 1 Check the version of your installed module.\nSo, firstly I have checked the version of the module I had locally. The name of the module is AzureRm.DataFactoryV2, so run:\nGet-Module AzureRm.DataFactoryV2\rModuleType Version Name ExportedCommands\r---------- ------- ---- ----------------\rScript 0.5.3 AzureRm.DataFactoryV2 {Get-AzureRmDataFactoryV2, Get-AzureRmDataFactoryV2ActivityRun, Get-AzureRmDataFactoryV2Dataset, Get-AzureRmDataFactoryV2IntegrationRunt... Then, let\u0026rsquo;s check the latest version fron the Internet:\nFind-Module AzureRm.DataFactoryV2\rVersion Name Repository Description\r------- ---- ---------- -----------\r0.5.10 AzureRM.DataFactoryV2 PSGallery Microsoft Azure PowerShell - DataFactories service cmdlets for Azure Resource Manager Seems like need to be updated (you must run a PowerShell session in administrator mode):\nUpdate-Module AzureRm.DataFactoryV2 OK, let\u0026rsquo;s try run the creation of ADF again:\nSet-AzureRmDataFactoryV2 -Name \u0026#34;KamilExp-adf2\u0026#34; `\r-ResourceGroupName \u0026#34;rg-somethingbad\u0026#34; `\r-Location \u0026#34;westeurope\u0026#34; I\u0026rsquo;ve received the error again, BUT with different, more valuable cause:\nSet-AzureRmDataFactoryV2 : HTTP Status Code: NotFound\rError Code: ResourceGroupNotFound\rError Message: Resource group \u0026#39;rg-somethingbad\u0026#39; could not be found.\rRequest Id: d051c049-8fb5-4e40-be7e-461c97904018\rTimestamp (Utc):09/12/2018 16:38:32\rAt line:1 char:1\r+ Set-AzureRmDataFactoryV2 -Name \u0026#34;KamilExp-adf2\u0026#34; -ResourceGroupName \u0026#34;rg ...\r+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r+ CategoryInfo : CloseError: (:) [Set-AzureRmDataFactoryV2], CloudException\r+ FullyQualifiedErrorId : Microsoft.Azure.Commands.DataFactoryV2.SetAzureDataFactoryCommand As you can see - the problem was trivial - I wanted to use non-existing Resource Group.\nThis can happen when you make just a typo or switiching between many subscriptions and used resource group that belongs to other subscription.\nHint 2 Check your contexts and what RG are available:\nGet-AzureRmContext\rName : Visual Studio Enterprise (******-5506-4104-8c84-0051fc36****) - Kamil.Nowinski@******\rAccount : Kamil.Nowinski@******\rSubscriptionName : Visual Studio Enterprise\rTenantId : ********-80ee-4819-a9ce-863d5afbea1c\rEnvironment : AzureCloud\rGet-AzureRmResourceGroup\rResourceGroupName : rg-datafactory\rLocation : westeurope\rProvisioningState : Succeeded\rTags :\rResourceId : /subscriptions/******-****/resourceGroups/rg-datafactory\rResourceGroupName : rg-dwh\rLocation : eastus\rProvisioningState : Succeeded\rTags :\rResourceId : /subscriptions/******-****/resourceGroups/rg-dwh\rResourceGroupName : rg-others\rLocation : northeurope\rProvisioningState : Succeeded\rTags :\rResourceId : /subscriptions/******-****/resourceGroups/rg-others Switch to appropriate subscription or change the [ResourceGroupName] parameter and run it again:\nSet-AzureRmDataFactoryV2 -Name \u0026#34;KamilExp-adf2\u0026#34; `\r-ResourceGroupName \u0026#34;rg-datafactory\u0026#34; `\r-Location \u0026#34;westeurope\u0026#34; If that\u0026rsquo;s doesn\u0026rsquo;t work for you - restart your PowerShell session. It turning out that even though your module seems updated - still might not work correctly. For example, reloading the module to previous version I had (0.5.3) is not possible:\n$m=\u0026#34;AzureRM.DataFactoryV2\u0026#34;\rRemove-Module $m\rGet-Module\rImport-Module $m -RequiredVersion 0.5.3\rException calling \u0026#34;Add\u0026#34; with \u0026#34;2\u0026#34; argument(s): \u0026#34;The key \u0026#39;Set-AzureRmDataFactoryV2:ResourceGroupName\u0026#39; has already been added to the dictionary.\u0026#34;\rAt C:\\Program Files\\WindowsPowerShell\\Modules\\AzureRM.DataFactoryV2\\0.5.3\\AzureRM.DataFactoryV2.psm1:61 char:9\r+ $global:PSDefaultParameterValues.Add($_,\r+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException\r+ FullyQualifiedErrorId : PSArgumentException Simply, restart the PowerShell session and the latest version of the module should works correctly.\nConclusion Once you notice that error:\nSet-AzureRmDataFactoryV2 : HTTP Status Code: NotFound\rError Code: NotFound\rError Message: Operation returned an invalid status code \u0026#39;NotFound\u0026#39; Do:\n1) update AzureRm.DataFactoryV2 module first.\n2) restart PowerShell session (if needed)\n3) check the correctness of passing Resource Group Name\nPlease bear in mind that the module is still developing (0.something version) and thus might contain some errors.\nI hope that helps and save you certain minutes/hours.\nCheers,\nKamil\n","date":"2018-09-20T00:59:37Z","image":"/uploads/2018/09/error-2129569_1280.jpg","permalink":"/2018/09/notfound-status-code-when-working-with-adfv2-via-azurerm-datafactoryv2-powershell-module/","title":"NotFound Status Code when working with ADFv2 via AzureRM.DataFactoryV2 PowerShell module"},{"content":"Unlock petabyte-scale datasets in Azure with aggregations in Power BI\nNow petabyte-scale datasets in Azure is possible - watch how aggregations feature in Power BI works.\nIntroducing Azure DevOps\nCall me Azure DevOps from now, said VSTS.\nWhen to use CmdletBinding in PowerShell?\n\u0026hellip;and how this is tied with Write-Verbose?\nCosmosDB 2.1.7.675\nThis module provides cmdlets for working with Azure Cosmos DB database, collections, documents, attachments, offers, users, permissions, triggers, stored procedures and user defined functions.\nPower BI Desktop September 2018 Feature Summary\nBuilt-in report theme options, Copy value and selection from table \u0026amp; matrix, and much more!\n","date":"2018-09-17T22:15:33Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/09/last-week-reading-2018-09-16/","title":"Last week reading (2018-09-16)"},{"content":"The SQLDay conference happened this year again. The biggest conference about Microsoft Data Platform in that part of Europe and the only one organises by community. The formula is the same: 3 days whereas first day is a workshop and another 2 days are full of regular session spread across 5/6 tracks. I had a pleasure to play a few roles: I was leading full day pre-conf, have had a regular (60 minutes) session and be co-organizer of the conference. But, let\u0026rsquo;s start from the beginning.\nPre-pre-conf As the conference has started on Monday, the team started its work the day before, on Sunday. Organisizing such big conference it\u0026rsquo;s always a challenge. Fortunately, the location hasn\u0026rsquo;t been change for a few years (Centennial Hall) which helps the preparation. Anyway, there is a lot things to do before conference starts, e.g. fold T-shirts and pack these stuff to goody-bags. Obviously, do not forget make a bite \u0026amp; take a sip of beer.\n\u0026ldquo;I have nothing to offer but\u0026hellip; \u0026hellip;blood, toil, tears, and sweat.\u0026rdquo; We’re setting the venue for tomorrow. #sqlday pic.twitter.com/ydwzO4CSwu\n— Kamil Nowinski (@NowinskiK) 13 May 2018 As always, it\u0026rsquo;s fantastic opportunity for me to meet my polish friends. That\u0026rsquo;s why I have joined them in the venue on Sunday. Even for a few dozens of minutes only.\nSo, I\u0026rsquo;ve said hello, have had a chat and couldn\u0026rsquo;t help too much. On Monday I had something important. Something I had to prepare to. This evening supposed to be even more busy for me this time.\nMy first workshop At the end of our workshop That\u0026rsquo;s right. I was to lucky to be selected to present my full-day session about DevOps with Microsoft\u0026rsquo;s database. As it was my first time, I asked Damian Widera (T) to present this session with me. I had a scenario for a workshop, we\u0026rsquo;ve discussed about execution (plan) and kicked off the preparation.\nWhole workshop contained 4 modules, so we divided the content by two - each of us prepare 2 modules. If you want to see the subjects we covered - check it out on this page.\nSo, the day before - we had to check our materials, demos, and refine all a bit more to be better understood by our attendees and remit the knowledge as smoothly and efficiently as possible.\nWe had 61 attendees in the room, which is very, very good result in my opinion. During my preparation for the workshop, I had 2 ideas. One of them was to prepare a short survey and ask our listeners about their opinion. That opinion was very important for me for a couple of reasons:\na) it was my first workshop, so I appreciate all feedback about my presentation and demo\nb) I\u0026rsquo;d have to understand how they found the workshop, which parts were useful and what can be improved.\nIn order to encourage the attendees - we have had books for draw out at the end of the conference. Both with signature of its authors (Itzik, Dejan)\nBriefly, these are the answers to the survey\u0026rsquo;s questions:\nThanks to all the people who took part in that survey. Much appreciate and also, I\u0026rsquo;m very glad that you liked it.\nMemorable Evening With my friend Rob Sewell I thought that the worse is behind me. I was wrong.\nAfter the whole fatiguing day, it was time to go to a hotel and rest\u0026hellip; no, forget. Check, makeup, practice my tomorrow\u0026rsquo;s session about Azure Data Warehouse. AZURE. Cloud. So, you need to have the Internet, valid subscription and enough money on it. As you probably know - Azure SQL DW is not cheap (toy).\nI always try to be well prepared for my session. It\u0026rsquo;s increasing my self-confident and decreases stress (yes, I\u0026rsquo;m still getting a stress before I present, but I\u0026rsquo;m calm that it\u0026rsquo;s common for most speakers and good sign - my friend Rob Sewell (T) has written post about it here).\nHence, first goal was to check the demo (scripts). Before I started doing so, just quick take a look at emails. And then I found email from Microsoft that my subscription has been disabled.\nOh no. It wasn\u0026rsquo;t funny. Not today. Microsoft, you have 362 days (365-3) in the year to do this and you have to choice one of those three? Great\u0026hellip;\nFirst thought: did I reach a limit of my subscription? Of course not! I mean\u0026hellip; 99% not. I\u0026rsquo;ve raised a ticket to the support explaining the situation and its seriousness (at least for me).\nSecond thought: I have second account, let\u0026rsquo;s try that one. When I tried to log in to the Azure portal - I\u0026rsquo;ve got the message like this:\nYour account has been locked.\nDamn it. What happens now? Is that due to my location? This account should not be used outside of UK. Anyway\u0026hellip; I have no active Azure account and tomorrow morning I have my presentation. What could I do? Nothing. Hence\u0026hellip; went downstairs and I\u0026rsquo;ve got a beer.\nSpeaker Dinner \u0026amp; subscription Thanks from Community To be honest, it was the best decision I could have made.\nFirst day\u0026rsquo;s evening was a speaker dinner at the event hotel. I went down there, grab some beer and start talking to people. And then, it turned out that I have more friends that I assumed. Janusz (T|B) offered me to swap our session and as he has had his session last slot day-after - that might help me to try sorting out everything. Help accepted, +10p to Janusz appreciation account and my many thanks!\nThen Michał Smereczyński (T) offered me a subscription that he had on the pocket. Wow! Many thanks! That\u0026rsquo;s becoming my session feasible again. The last concerns which left were about the data (flat files) I had prepared in my blob storage. \u0026ldquo;Gonna be fine. I will come up with an idea tomorrow. Today, just focus on community. Eat something, drink a little bit and have a good time with friends\u0026rdquo;, I have thought. And I did it.\nSecond day It was great that I had enough time to check and repair my presentation.\nMeanwhile, Microsoft\u0026rsquo;s support replied to me and enabled my subscription. Unfortunately, they couldn\u0026rsquo;t ensure that it will not be disabled again automatically. I couldn\u0026rsquo;t take that risk. I used AzCopy to copy all my demo CSV files and double check my demo. I was ready. The crisis has been defused.\nAnd I spend most of the day in the speaker room.\nThe party! As usual on the evening of the second day - the party is happening. There is a good music (my good friend Mateusz (DJ MTM) played DJ role), plenty of various food, bunch of drinks to choose and many opportunities to have fun and good conversations (including our fantastic speakers). So, what is a good enough reason not to join to such party?\nCharity When you giving a regular, 60 or 75 minutes, session - you are doing it as a volunteer, for free. And this is fine for all of us, even though it does cost a lot of efforts to prepare a quality session. The longer session is the most effort, work and time you need to spend on it. Typical, isn\u0026rsquo;t? So, this is not a secret but maybe not everyone know that the workshop sessions are payable to the speakers. This may vary depends on conference, of course, but generally, that is the rule.\nWhen I was preparing the workshop I came upon an idea. Don\u0026rsquo;t understand me wrong. As that would be extra money for me - why not to share the money with someone who need it more? Always there is some people, children, animals or charities whom that money can help much more than me, right? I believe that the karma returns.\nIt worth to mention that I was also inspired by two people: the polish lead financial blogger, Michal Szafranski (T) and my UK friend - Alex Yates (T). Alex has given away ample amount of money to two charities several months ago (read about it here) and Michal regularly helps one of polish charity who feeds malnourished children.\nSo, my idea was to spend part of earned money on children. Hence, I decided to start with 10% which is maybe not a break-taking sum, but it still better than nothing, right?\nI shared this idea with Damian (we led the workshop together) and he agreed to do this. I can\u0026rsquo;t even imagine how it\u0026rsquo;s possible that some children can be malnourished.\nEventually, I wanted to reveal a confirmation of money transfer sent to \u0026ldquo;Pajacyk\u0026rdquo; as I have nothing to hide and simply can confirm that I did it:\nmy part of \u0026ldquo;better than nothing\u0026rdquo; WhatIf I did it. Very easily. Without any problems. And I felt happy. Again: don\u0026rsquo;t get me wrong: generally I\u0026rsquo;m a fan of giving people fishing rod rather than a fish and I\u0026rsquo;m not going to discuss about these things deeper. BUT.\nWhat if at least part of us has possibilities to spend at least part of that extra earned money to freely selected charity? Just think about it: how many of us do the workshops across the world? How many conferences we have got every month, every year? That would be massive help!\nNaturally, everything is up to you. This decision is thoroughly under your discretion. And of course only when it\u0026rsquo;s not your only one source of income.\nHowever, if you would decided to do that - spread a word on twitter tagging it with #sqlcharity and name/handle of organisation you have supported.\nLet\u0026rsquo;s make a new trend. Let\u0026rsquo;s make something big. Let\u0026rsquo;s change the world. Even though a little bit. Perhaps it saves somebody life?\n#sqlcharity can help. We can help.\nClosing session At the end of the conference, there have not run out of a raffle. As only in good events - only present people can take part of the raffle. This year we had a several special prizes and tons of books. Of course - Szelor\u0026rsquo;s contest has taken place either. Interested what is it? I will not reveal that mystery - you need to come and experience yourself.\nThanks to my friends, Damian and I, had opportunity/capability to giving away 2 books to attendees of our workshop. The books contained the autographs from Itzik Ben-Gan (T) and Dejan Sarka (T).\nSQLDay in numbers This year reach the limit. 800 people is the maximum what the venue can afford. That\u0026rsquo;s why we had to close the registration 2 months before the event.\nIt shows us that the event enjoys great popularity and it\u0026rsquo;s our gratification. Below you can find out some figures:\nWhere attendees come from? Gender Companies Attendees Women: 14.2% Men: 85.8% 282 Workshops (only): 4% Conference (only): 36% W+C: 61% Why it is worth? As I mentioned, it\u0026rsquo;s biggest Data Platform conference in Poland (and probably in that part of Europe) with many international speakers which means two things: very good quality content, session in English as well (we try to have 50% of them).\nValue for money The prices for entire conference is really really reasonable! Only 1000 PLN (about 250 EUR) is almost nothing comparing to ton of knowledge you pick up during the event. When it\u0026rsquo;s still out of your reach - ask your line manager or boss to send you onto such training. When even this scenario fails - why not to be volunteer on SQLDay? Let us know: sqlday@sqlday.pl\nWhen will be the next conference? Next event will take place in Wroclaw, Poland between 13-15.05.2019. We already know big names of this conference:\nAaron Bertrand (T) Grant Fritchey (T) https://www.youtube.com/watch?v=g-nmExw35Y0\nYou don\u0026rsquo;t have to rush yet. The registration will commence December/January, so stay tuned and buy your ticket in the best prices. Early bird last only first few weeks. We don\u0026rsquo;t know exact prices yet, but certainly they will be very similar to this year.\nCFS buddy. I wanna be a speaker! You\u0026rsquo;re more than welcome to submit your session. CFS doesn\u0026rsquo;t start yet and follow twitters: @SQLDay and/or @SQLPlayer if you want to be up to date with the news.\nGreetings I want to thanks to all volunteers, speakers and organisers. Data Community Poland is the organiser of this conference and it\u0026rsquo;s a group of people who are professionalist and passionate in their work. Big, big thanks to you all, my #SQLFamily.\nPictures fot. Dariusz Brejnak\n[srizonfbalbum id=10]\n","date":"2018-09-10T15:46:15Z","image":"/uploads/2018/09/I7D_8463_SQLDay2018_mid.jpg","permalink":"/2018/09/sqlday-2018/","title":"SQLDay 2018"},{"content":"Automatic intelligent insights to optimize performance with SQL Data Warehouse\nWith no additional costs, Azure SQL DW delivers you best practice recommendations in terms of data skew and suboptimal table statistics.\nNew monitoring capabilities for Power BI Premium Capacities\nLooks as quite cool way to monitoring your Power BI.\nSingle-Column-Key Missing Index Recommendations are Usually Wrong\nAh, it\u0026rsquo;s very common. But if you still learn about indexes - check why not worth (in many cases) taking SSMS\u0026rsquo; advises seriously and asking sp_BlitzIndex instead.\nThe August release of SQL Operations Studio is now available\nNew version of SOS has been released. Check how easy and smart the process of import data from flat files can be now.\nEvent: SQL Relay\nWhy not come to this free conference? It will take place in 5 cities in the UK in 1 month times. I\u0026rsquo;ll be there all week long. You\u0026hellip;?\n","date":"2018-09-09T23:29:49Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/09/last-week-reading-2018-09-09/","title":"Last week reading (2018-09-09)"},{"content":"I would like to spread the word about a new conference which appears in my favourite city in Poland - Wroclaw. I\u0026rsquo;m talking to Jacek Biały, Business Intelligence Competency Center Manager in Sii, who shed the light on that event for us.\nKamil Nowiński: Which conference from the cycle and when did the idea of its come about?\nJacek Biały: It’s the first edition of BI Connected, but not the first our conference. Sii twice a year organizes Get.Net – one of the biggest .Net conferences in Poland. We have also organized the conference dedicated to Scala developers and the one for Angular developers.\nThe idea of organizing BI conference came to our mind at the beginning of the year. Business Intelligence is very important area in Sii and we feel that this kind of event is needed in the BI community.\nKN: Why Wroclaw, actually?\nJB: Wroclaw is the new Silicon Valley with more than 12000 IT specialists, working in 200 companies. Part of them focus on Business Intelligence as an integral part of business development. BI community is also very active in Wroclaw. On the other hand our branch in Wroclaw is also highly developed in BI area. We have many BI projects run directly in Wroclaw. All this factors make Wroclaw great place for that meeting.\nKN: How many sessions are planned?\nParticipants of the event will be able to listen 9 speeches made by BI experts. The conference topics include among others Power BI, SQL Server, Data Vault 2.0 and Qlik Sense Cloud.\nKN: When does the registration for the event start?\nJB: Registration and ticket sales started at the end of August. All the information can be found on biconnected.evenea.pl\nKN: For those who are just looking at the labor market or do not yet know the concept of BI - let\u0026rsquo;s say what Business Intelligence is today, what areas does it cover and what technical tools are used to develop BI class solutions?\nJB: Simplifying the definition of BI is: “Business Intelligence transform raw data for business purposes friendly”. In a one word, all actions aimed at transforming data into understandable business information is BI and the question is what action we can take now contribute to the transformation of data? Nowadays such activities and processes are all kinds of ETL processes (Extract, Transform and Load), construction of data warehouse or Data Lake, statistical data modeling, data visualization (reports, dashboards), which allow us to extract essential knowledge from the data. To do this we need the right tool support and that\u0026rsquo;s what we get. Currently, we have a lot of tools available for processing data such as: Power BI, Qlik Sense, Informatica Power Center, SAS, SAP BO or various tools from the Big Data area: Hadoop, Spark, Kafka, Scala, Hive and many others. I don\u0026rsquo;t describe these tools here and encourage everyone to check them out for themselves, but I would like to invite everyone to explore the amazing BI world, where you never know what secrets you can discover.\nKN: Thank you and I wish you good luck also with this conference.\nJB: Thank you.\nUseful links: BI Conference: Registration | Facebook page\nSii on LinkedIn\nSii website\nGet.Net conference\n","date":"2018-09-04T22:26:24Z","image":"/uploads/2018/09/BI-Conneted_grafika.jpg","permalink":"/2018/09/bi-conference-in-wroclaw/","title":"BI Conference in Wroclaw"},{"content":"Good day SQL folks!\nHere we are, new week, new Monday and something new to read about the Data Platform world.\nTLA+ specification of the bounded staleness and strong consistency guarantees\nThe 8th of many Technical Blogs on Azure Cosmos DB database internals - TLA+ specification of the bounded staleness and strong consistency guarantees\nClustering on mixed type data\nClustering unsupervised data is not an easy task. Indeed, data crunching and exploration is in such a context often driven by domain knowledge, if not pure intuition, and made difficult as there is no way to measure the accuracy of the resulting segmentation (as opposed to supervised learning).\nWhat the Shard? Partitioning Azure Cosmos DB with the Mongo DB .NET Driver\nAzure Cosmos DB is a fully managed NoSQL database. An advanced and powerful feature is the ability to partition data. This is commonly referred to as sharding.\nConfigure a SQL Server container in Kubernetes for high availability\nLearn how to configure a SQL Server instance on Kubernetes in Azure Container Service (AKS), with persistent storage for high availability (HA). The solution provides resiliency. If the SQL Server instance fails, Kubernetes automatically re-creates it in a new pod. AKS provides resiliency against a Kubernetes node failure.\nOn-premises data gateway August update is now available\nRead about the improvements to the custom data connectors support announced in the July release. With the August release, the enterprise gateway now supports custom data connectors that are using the Direct Query technology and\\or using ODBC drivers.\nASF 013: James Rowland-Jones interview\nJames Rowland-Jones is a Principal Program Manager for Microsoft. He is passionate about delivering highly scalable solutions that are creative, simple and elegant in their design. James specializes in big data warehouse solutions such as Microsoft Analytics Platform System (APS), Azure SQL Data Warehouse, Hadoop ecosystems\nCheers,\nDamian\n","date":"2018-09-03T04:34:12Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/09/last-week-reading-2018-09-02/","title":"Last week reading (2018-09-02)"},{"content":"Processing Petabytes of Data in Seconds with Databricks Delta\nDelta is a unified data management system that brings data reliability and performance optimizations to cloud data lakes.\nMonitor Azure Data Factory pipelines using Operations Management Suite\nNow you can send your data factory metrics to Operations and Management (OMS) Suite. Check how to achieve that (video).\nAzure Storage, How fast are disks?\nMy teammate Ed [T] has done a very well investigation and tested the productivity of various discs in Microsoft Azure. Check this out if you need Azure VM with very high IOPS or simply check what is enough for your needs.\nAzure SQL Data Warehouse Gen2 now generally available in France and Australia\nHere you also have a full map of Gen2\u0026rsquo;s availability.\nPython Episode 1 – A New hope\nPython is dancing with Power BI. Many of us were waiting for that feature!\nHow to cause a simple spill to tempdb\nKendra Little [T] shows a quick demo sample to cause a spill. It works everywhere, not only in huge databases.\nSQL Relay conference is coming\n5 days, 5 conferences. Absolutely free! Registration opening soon!\nJoin us for a week of awesome free training on the Microsoft Data Platform.\rRegistration opening soon! Join our mailing list https://t.co/8667S4UOeP for event news and updates. pic.twitter.com/mcJYJBPCID\n— SQL Relay (@SQLRelay_uk) 26 August 2018\r","date":"2018-08-28T23:26:39Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/08/last-week-reading-2018-08-26/","title":"Last week reading (2018-08-26)"},{"content":" Introduction James Rowland-Jones is a Principal Program Manager for Microsoft. He is passionate about delivering highly scalable solutions that are creative, simple and elegant in their design. James specializes in big data warehouse solutions such as Microsoft Analytics Platform System (APS), Azure SQL Data Warehouse, Hadoop ecosystems.\nJames is a keen advocate for the SQL Server community; both internationally and in the UK. He has previously served on the Board of Directors for PASS and helped organise SQLBits. James was awarded Microsoft’s MVP accreditation from 2008 - 2015 for his services to the community.\nThis talk has taken place during SQL Bits conference in London, on 23rd February 2018 (Friday).\nInterviewers: Kamil Nowinski \u0026amp; Damian Widera (remotely).\nDo you want to know why James has moved to Seattle and started working for Microsoft?\nHow he started his career in IT and what is important in your work? Which big conference is his baby and how much his community involvement has affected his professional life?\nYou will find out all the answers in the following talk.\nTranscript * Bear in mind that term \u0026ldquo;compute optimized\u0026rdquo; for Tier 2 is no longer in use.\nKamil Nowinski: Our today’s guest is a Principal Program Manager of Microsoft since December 2015, former Director at Large in PASS for almost four years, former Organizing Committee Member for seven years and currently part of data platform teams supporting Massively Parallel Processing workloads in hybrid architectures. When you are working with Azure SQL Data Warehouse, you’ve heard about him very lightly. So our today’s guest is…\nJRJ: James Rowland Jones.\nKN: James, where do you live currently?\nJRJ: I currently live in Sammamish which is just outside of Redmond and just past Lake Sammamish in Washington state.\nKN: It is very close to Seattle, right? To Redmond basically?\nJRJ: Yes, it is very close to Redmond.\nKN: However, you used to live in the UK. Why do you decide to move to the USA?\nJRJ: I’ve been working in consulting for many years – works in a partner eco-system for a long time. I realized that my impact was ultimately one customer at a time and I could do my best for that one customer. But then the opportunity came up for me to be able to transfer over and actually do whatever I could to help everybody all at once, so I can release one feature and help everybody. The opportunity to be a part of the team and making the product and be a part of the journey that we’re going on with Azure SQL Data Warehouse was too good to pass up. My family and I had some very long conversations. We decided that we want to do it. And that was it. We moved over just a year ago. Here at SQLBits this is my first time back. It is great to be here and see the SQLBits carrying on. I’m going from Strength to Strength and I can’t begin to tell you how proud I am of this event in terms of what it means to the global Community of data professionals. Even at Microsoft I can see an echo. Everybody cares very much about the Community and the success that we were able to achieve with SQLBits. It’s great to see it continuing and long may it do so.\nKN: You have been with SQLBits from the beginning?\nJRJ: Pretty much. I first came over to SQLBits as an attendee. I attended the first one. And at the second SQLBits I have spoken for the first time I have ever spoken about anything. I did a talk on I/O. And then the third one I started to get involved and at the fourth one I started to be actively involved in the Committee of SQLBits. And then pretty much what happened to the point where I left and joined Microsoft, I was working very closely with Simon, Darren, Aaron and Chris – all of the organizers over the years to make these events happen. This is one of the things I probably would say I am most proud of in all of my professional carrier. It has obviously in many ways nothing to do with my professional carrier because it is all volunteer. Like yourself, we would kind make it happen. It is a great thing. Something that was truly special.\nKN: It is good to see how your child is growing.\nJRJ: Absolutely.\nDamian Widera: James, what are you doing for a living?\nJRJ: I’m a Principal Program Manager at Microsoft now and I work on Azure SQL Data Warehouse. Up until very recently, I was actually working directly on product focus. And the current preview of the compute optimized* of the performance tier was the area that I was leading. And I was delighted to be able to get that into a preview. Just recently, like a week or so ago, I have slightly different role within the team. Now I lead a team of PMs and we look at the end to end ecosystem around Data Warehouse in analytics in general. So thinking about what SQLDW role is an analytics platform but also thinking about what are they associated, related services, how do they integrate together, how do we make sure…\nKN: So you have a little bit wider view currently…\nJRJ: Yes, much broader perspective on how we actually make the magic happen at the Azure.\nKN: All the integrations.\nJRJ: Yes, some of it is a big integration, some of it is making sure that we have clear patterns and guidance – stuff like that.\nKN: I’ve been on your session where you have been showing the new generation of Data Warehouse in Azure. Could you tell us a bit more about it? What new is in the new generation of Azure Data Warehouse?\nJRJ: This compute optimized* performance tier which we have is like the next generation of SQLDW. It introduces new hardware from Azure. It is new NVMe’s SSD capability into the compute node itself. And with that, is means that we can add to our story around separation of computer storage this intelligent cache locally in the computer nodes which eliminates beneath remain I/O when querying large volumes of data.\nSecond thing – to make use of that cache is that we externalize all of the columns to segments and we hold them independent of the actual SQL server data files. So you can imagine how impossible this would be in an on-premise world. We’re taking that data and we’re holding it in the remote storage. And then – as customers query their column-store data, we move that column-store segment up into the intelligent cache and we manage it there so that queries – especially weak queries – run exponentially fast.\nKN: And the cache is pretty amazing – 400 GB.\nJRJ: The nodes themselves are quite large physically. Every node today is about 400 GB more of physical memory and then you basically come down to the cache itself and the NVMe SSD disk space cache has about 1.5 TB of cache in that node and that is actually what delivers the hottest data in the warehouse which you can imagine in Data Warehouse obviously like tens of the tail of the table, keeps that nice and tight to the CPUs use so that we can really deliver the best in class performance. It’s very exciting to see that.\nKN: It’s 100 better improve performance, right?\nJRJ: Yeah. My favourite individual customer experience that had one of our specific customers were able to get some of that queries to run kind of 100 times faster but also they were running with half the compute costs. They literally were running half the compute power but got this massive performance benefit on some of their queries. Which is really great to see. That is a great value for customers.\nKN: Yes, it is. You mentioned also the data movement service.\nJRJ: One of the things that should be in the system is that in order to get the resolve to query you have to be able to transparently move data across the nodes in order to make it works properly. So what we do to achieve that is we use the technologies like the data movement service. But what we’re doing right now is actually that we took some of the core data movements type – one in particular called the shuffle move – and we made it an in‑process database movement and because of that we’ve also seen phenomenal performance improvements because the data itself just moves that much faster around the system.\nKN: Those things are all amazing and probably the customers will be asking when it will be generally available. [this talk has taken place on February 2018].\nJRJ: We’re currently targeting for Spring of this year [2018] to bring it to the general availability. Where we are at right now customers can and are using the preview in about six regions of Azure and we’re aggressively onboarding new regions as we go. Azure SQL Data Warehouse currently is available in 33 regions in the world. I think it is over double that any other Cloud Data Warehouse available in the market today. It’s a great story for us and it’s a very important product.\nKN: And you are as an employee and very important person from that perspective – from Data Warehouse in Azure – you are working with customers very closely. And from that perspective, how do you think how fast customers are adopting those new technologies in their environment?\nJRJ: These are some very interesting patterns, actually. What we have seen in a lot of the time was that customers moving into Data Warehousing in the Cloud really wanted to take advantage of what the Cloud can offer them rather than just pick it up and just flip it into the Cloud. They wanted to see what they can really do to modernize the platform, they’re separating out their data, they’re trying to leverage also like a data lake driven approach in many cases. Sometimes they don’t even realize they’re doing it, the way as they transfer their data into BLOB Storage and then move the data through into the DW. So we’ve seen customers of all shapes and sizes across all the major verticals which is great to see. It’s not just kind of landed in one vertical. That’s always very important for us. And then the other thing that we’ve seen is that you get these new styles of workloads where customers are integrating basically like if you can think about transactional feed with your Cloud data sources – like dynamic CRM or Salesforce – whatever – integrating that into that flow and then also maybe even some sensor data IoT style. All those different types and flavours of data, all coming in and then using the DW to serve that out organizationally. It’s interesting because the more typical workloads and then you get the people to see the potential doing something different and then even going on and building out some new capabilities. It’s been great to see customers integrating ML [Machine Learning] modules over their data in the DW doing things like that to actually take advantage of all the connected services available in Azure.\nKN: If they move, they already see new possibilities.\nJRJ: Absolutely. That’s a very real thing for people.\nKN: Also those new features like Graphical Execution Plan in the newest SSMS. I think it’s a very cool direction. But how much can it help right now?\nJRJ: The important thing we found when people were using Explain plans in the old way is that it took quite a lot of explaining to explain how to use the Explain plan – that’s quite a tongue twister – but it meant ultimately for us that we were making our customers lives more difficult. We wanted to have something that we give them like the conceptual view and visibility of the plan in a way that people knew and understood and were happy with. It’s great to see the overwhelmingly positive feedback we’ve had when I’ve shown the use-case both in my Pre-Conf with the latest on Wednesday and also when we show it in the session today. That’s been fantastic.\nKN: But the plan is not producing by the engine itself.\nJRJ: Yes, it’s producing by the MPP engine.\nKN: Ah, is it?\nJRJ: Yes. So literally what you are seeing when you run the Graphical Explain Plan is exactly the approach that had you run the query that is what SQLDW would have used as a plan to actually execute your query. So, it’s a real-real.\nKN: Has it been finished or would it be extended?\nJRJ: I think we’re at the point when we’re happy with what we’ve got, we want to see what customers want and what feedback we get now. We’re more in the actively listening mode again for that particular feature.\nDW: What do you think - what will be the future of the cloud?\nJRJ: I think that the future of on-premises is more questionable than the future of the Cloud The future of the Cloud has a pretty clear message. I mean the vibe here in SQLBit since I’ve been here, obviously doing this for many years – the question is not like “Should I move to the Cloud?” or “When can I move to the Cloud?” but the question is: “How? How do I get there? What do I need to do?” All of those sorts of questions. I don’t see it as a question about the future of the Cloud. I think it’s more about how customers are moving their workloads to the Cloud, how quickly they can do that and what’s driving that. And then for the workloads – it’s stay on-premise and there are clearly legitimate reasons for that. It’s about who can support you best there and what customers solutions do we provide in that space. And that’s what it’s exciting about working in Microsoft – we support customers where they’re at with their requirements. And I feel that we have some very competitive offerings whether you’re on-premise or in the Cloud.\nDW: Yes, that’s right. And the question could be in the future of course why am I still not in the Cloud? I should be there because that’s the future.\nHow do you start working with data?\nJRJ: I actually have a law degree. I’m not a computer scientist in any stretch. I happened to been sat at home very unemployed one day. I was watching a TV show in the UK called Working Lunch. It was a lunchtime business show. And it was in about 1998. There was a chap on TV. They were interviewing him from LOGICA – now the CGI is the current version of this company – they were talking about an international phenomenon known as Y2K and they were talking about the shortage of technical resources to solve the problem of Y2K compliance and coding. CGI’s response for that at that time was a 4-week training course as an apprenticeship. And if you turned up at the university and pass the interview, they would put you on this 4-week training course and turn you onto the Y2K programmer. That’s how I got into it. I actually turned up. I could honestly tell you that I failed the math test but I managed to Ace the interview and they let me in. And the rest is history. Five weeks later I was having my first interview and then I spent my first two years working on Ford of Europe’s mainframes making Y2K changes to their vehicle accounting system.\nKN: Wow! That’s amazing history.\nDW: And now you’re at Microsoft so what is the most important part of your work?\nJRJ: I think wherever you go the most important part of your work is the people who you are working with and actually making sure that everyone is aligned to the goals that you have and they are actually working together to achieve those goals. When you have that alignment and that energy and everybody is pulling together that’s a phenomenal feeling. It’s great to see things land – you win together, you lose together. That’s a very important lesson not just professionally but also in live.\nKN: Working as a team.\nOk, can you tell us how are you preparing for a speech exactly?\nJRJ: It’s really thinking about the audience. What do you think people need to know? What outcomes do you want them to have from having attended your session? One of the things about coming to this is like there is the wealth of choice so appreciating the people who choose to attend your session is very, very important. And once you have that, the next piece is really about making sure that you give them something that is somehow actionable. Sometimes knowledge for knowledge’s sake has very limited value in my opinion. Whereas if I give you something with the purpose that you can go and do or you can go and try, or you can explore and you take it forward, that tends to have much more meaningful impact overall. But just me telling you about my product, you’re just saying that it was really interesting and we all go away that doesn’t necessarily change anything for anybody. That’s why I try to think about showing things like we showed three things today in the session which haven’t been really shown publicly before but these are the three things that people can go and do. And they can go and do them immediately and they can improve their solutions when they go back to work. One of the most important things when SQLBits were founded was the experience which the speakers had: Was is the real-world experience? Did they were able to communicate the value and actually help people who were attending with their everyday problems?\nKN: Show the practice from the real field.\nJRJ: So I try and do stuff that’s more true to that spirit.\nKN: How much important is passion for you?\nJRJ: I think personally it’s very important. You have to care about what you’re doing. There’s lots of people that are very passionate in this space and it kind of shines through. When you’re speaking about something you have to really care about it and you would have to able to communicate the value of it and I think it helps make your sessions come to live.\nKN: This podcast is not only about the technical stuff but also about the personality. How about your work-life balance?\nJRJ: As an interesting one – I’m sure if you asked Mrs. Rowland Jones, she would give you a very different answer. I work pretty hard I have to say in that regard. There’s lots of things that I care about: family, children – not really very small children but their still my children, nonetheless. Obviously, I love and care about them very much. I would say things that I can do with them are obviously very important to me. I love also my sports. For my sins I did go and see Arsenal play last night and that was fairly miserable experience but I couldn’t come all the way back to London and not see them play even if they were quite horrible yesterday. But you can’t have everything.\nKN: True.\nDW: I have a question about MVP program. What do you think about it right now? How important MVPs are for Microsoft?\nJRJ: I don’t know which forms people listen to but when I was an MVP, one of the things I valued the most as an MVP was the NDA alias that MVPs have access to. Likewise, I look on that alias as an employer of Microsoft and the feedback I get from that I take very seriously. I think MVPs have a very unique insight, a depth-level of your product and that it is in itself valuable. Clearly, they don’t speak for every person because they are specialists and probably in many cases the top 0.1% of professionals in the world and so their perception is normality of what’s normal everyday problem versus to the standard of which their code is like exponentially higher. You have to obviously take that into consideration as you balanced at the feedback. The feedback that we get especially from the SQL MVPs generally is very valid, very well respected but like I said – I’m giving you the broader context. You have to put it into the context that this is an absolute expert in the field of what they’re doing. Or it’s a very influential community person maybe speaking for a large group of people. You need to think of it in terms of those perspectives.\nDW: A little bit different question – what hints would you give to young people who want to start working on the IT market?\nJRJ: That’s a good question. I would say – getting involved. For example, I know where my career was at when I first got involved in the Community and I know where my career is now.\nKN: That’s why we’re asking that question to very experienced people.\nJRJ: And I owe an enormous amount to Simon Sabin who first introduced me to the Community and into being more of a volunteer and actually giving up time. I can honestly say I would never be able to pin point the specific ROY in doing a specific thing because it’s largely filled with intangibles. I guess at some point I just gave up carrying about that and I cared more about what I was able to achieve and what we were able to contribute. At the same time, I found that my passions took me in the direction that has obviously benefited me both personally and professionally. Without the Community investments I don’t think I would be where I am today and have the role that I have today. But in terms of just starting out, I think getting involved, going to use group meetings, going to SQL Saturdays is an integral part. But I think you should actually try stuff, being involved, be passionate about things and actually try to use technologies. I think that one of the things that a data professional is more of a boarding horizon and looking into new technologies and how they complement their already deep technical knowledge is definitely a thing.\nKN: It’s very valuable spot because I spotted that some young people has got that approach: „Oh, you know, it’s SQL Saturday but I’m not going because it’s my weekend.”\nJRJ: When I used to run my company called the Big Bang Data company which I run before I started my data profession and I used to take my son and he used to be on the stand with me on SQL Saturdays and he could deliver the pitch about SQL Data Warehouse as good as any Microsoft sales person that you would have ever meet. It was a lot of fun. I was very proud of him and seeing him getting involved and engaged with the Community and getting involved in a family business. I’m not sure that IT or life of data professionals is what he ultimately wants to do but even so – I was very proud of him for being involved and actually getting immersed and experiencing it for himself.\nKN: Thank you very much. At the end of our conversation tell us where we can find you.\nJRJ: So the obviously ways to find me is Twitter. You can get me also on LinkedIn. These are the two main social media platforms. But also I try to contribute to both the documentation for Azure SQL Data Warehouse as well. So when you are actually commenting on our docs and giving us feedback on things, you are connecting directly with the PMs and a writer as well. I think that’s a very important point and it’s not one to be forgotten when we think about how we engage with one another.\nKN: Ok, thank you again for this conversation. I really appreciate it.\nJRJ: Thank you. Take care.\nUseful links: James\u0026rsquo;s twitter: @jrowlandjones\nJames\u0026rsquo;s profile: LinkedIn\nJames on Channel9\nConferences: SQLBits\n","date":"2018-08-25T09:58:47Z","image":"/uploads/2018/08/IMG_9258_ASF013_SQLBits_with_James_Rowland_Jones_1200.jpg","permalink":"/2018/08/asf-013-james-rowland-jones-interview/","title":"ASF 013: James Rowland-Jones interview"},{"content":"Hi folks. I\u0026rsquo;m on holiday for 7 days (14 days left). The holiday is a perfect time for reading, right? Although I\u0026rsquo;m heading to Italy and writing this post in the car (don\u0026rsquo;t worry, my wife is driving), I prepared some interesting readings from passing week and added two new once today. Have a good week!\nCreating Sparklines And Small Multiples In Power BI Using The Google Image Charts API\nChris Webb (t) presents this quite interesting trick, albeit don\u0026rsquo;t use that solution on Production.\nDon’t do these things in SQL Server\nMy favourite part of this post and which I totally agree is \u0026lsquo;don’t use BETWEEN\u0026rsquo; (for dates)\nIntroduction to PowerShell with SQL Server Using Invoke-SQLCMD\nLearning PowerShell? This post is specifically useful if you want to migrate your batch script(s) into PowerShell.\nWhy Does My Select Query Have An Assert?\nUsually, an assert is being happened when you modify something.\nHow I Configure SQL Server Management Studio\nInterested how Brent Ozar (t) does configure his SSMS?\nExecute Jars and Python scripts on Azure Databricks using Data Factory\n10 minutes of video shows how to use Databricks script components in Azure Data Factory\nMassive list of video sessions. Worth remember it. Find out more here on the blog.\nSQLBits - sessions and content\nPASS TV\nGroupBy conference\n","date":"2018-08-19T18:07:29Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/08/last-week-reading-2018-08-19/","title":"Last week reading (2018-08-19)"},{"content":"Good day SQL folks!\nHere we are, new week, new Monday and something new to read about the Data Platform world.\nPower BI Desktop August 2018 Feature Summary\nThe biggest news from the last week was about the Power BI. There are a lot of new features but integration with Python could be the one we have been waiting the most (but not only).\nOptimization Thresholds – Grouping and Aggregating Data, Part 5\nGreat article written by Itzik Ben-Gan about query optimization\nAzure Tips and Tricks Part 147 - Run TSQL on an Azure SQL database with Azure Functions\nGreat blog of Michael Crump who works for Microsoft. He gives a lot of great examples of code in his blog.\nThe First 3 Things I Look At on a SQL Server\nAs the title states. What should be the first 3 things to look at on SQL Server according to Brent Ozar.\nFinding duplicated data in a case insensitive column\nInteresting post by Louis Davidson about finding duplicates in a very special case\nCheers,\nDamian\n","date":"2018-08-12T23:59:08Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/08/last-week-reading-2018-08-12/","title":"Last week reading (2018-08-12)"},{"content":"Does this video tell you anything about our next guest?\nThe next episode is almost here! Stay tuned \u0026amp; have a good weekend!\n","date":"2018-08-11T09:19:57Z","image":"/uploads/2018/08/ASF013-video-invitation.png","permalink":"/2018/08/asf-013-video-invitation/","title":"ASF 013 video invitation"},{"content":"Microsoft Azure Data welcomes Data Platform Summit attendees\nRohan Kumar [B|T] invites to have a look at today’s mobile-first, cloud-first world and what has been changed in the area by Azure Data team.\nAutomated Script-generation with Powershell and SMO\nOld enough (but still handy) post how to leverage PowerShell with SMO in order to generate a script of database DLL.\nA successful Git branching model\nA way of managing branches in Git is not such obvious, specifically when you\u0026rsquo;re beginning with it.\nExecute all SSIS packages in parallel with ADFv2 lookup and FOREACH activities\nAs ADF version 2 supports expressions and parameters, Paul [T|B] shows more dynamic method of run SSIS packages in the SSIS Integration Runtime (IR).\nMy thoughts on session levels\nIt\u0026rsquo;s not mine - it\u0026rsquo;s Hugo [T] thoughts and definitely worth to understand what the levels are. No matter whether you have just started attending on events or you\u0026rsquo;re a regular attendee and want to verify your assumptions.\nT-SQL Tuesday #104 Roundup\nFind out Bert\u0026rsquo;s [T] recap of 104th T-SQL Tuesday where the topic was \u0026ldquo;What code would you hate to live without?\u0026rdquo;. Certainly, you\u0026rsquo;ll find out tons of useful scripts.\nBuilding a data model to support 1 trillion rows of data and more with Microsoft Power BI Premium\nIn this video, Christian Wade [T] presents incredible new feature in Power BI: Dual model which allow combining Big data and Dimensional modelling scenarios.\nUsing that technique you can have import mode for some tables, Direct Query mode for the others and even create a virtual table with aggregated values.\nAmazing! I\u0026rsquo;m truly waiting for this feature as it\u0026rsquo;s not fully released yet.\n","date":"2018-08-06T06:34:47Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/08/last-week-reading-2018-08-05/","title":"Last week reading (2018-08-05)"},{"content":"SQL: More on finding rows that have changed using HASHBYTES and FOR JSON PATH\nThe title explains everything - Dr Greg Low shows it in practice.\nLondon Continuous Delivery - learn about Continuous Delivery\nMany materials about CD.\nMy References: Database DevOps Anti-Patterns\nCheck this video out where Alex explains anti-patterns in DevOps for databases.\nNew eBook - Data Lakes in a Modern Data Architecture\nMelissa Coates presents new version of their book where they updated 90% of the content. That\u0026rsquo;s the price of writing about things in Azure.\nAzure SQL Data Warehouse Gen 2: Adaptive, Concurrency and Scale\nMore about Gen 2 from Pragmatic Works.\nOverview of Data Integration Platform October \u0026lsquo;18 release\nNumerous of amazing features and improvements will be generally available in October\u0026rsquo;s release.\nCommunity recognition\nRandolph shares his experiences from being a member of the #SQLFamily, and how MVP award impacts on his life.\nMicrosoft Business Applications recorded sessions\nThis big event took place this week and you simply not miss anything having this link.\nDo you know Guy in a Cube? Adam shows you how to start with a data in Power BI.\n","date":"2018-07-29T22:31:21Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/07/last-week-reading-2018-07-29/","title":"Last week reading (2018-07-29)"},{"content":"Good day SQL folks!\nNow and it is almost Tuesday morning, let me give you some things to read for the rest of the day!\nExecute all SSIS packages in parallel with ADFV2 looup and foreach activities\nFollow up from the SQL Saturday in Manchester where Paul Andrew demonstrated how to use ADF with expressions and parameteres.\nStorage mode in Power BI Desktop (Preview)\nIn Power BI Desktop you can specify the storage mode of tables, providing you with control over whether table data is cached in-memory for reports.\nSimpler, smarter transformation and integration of enterprise data with Power Query and Data Integration Platform\nPower Query gets simpler and better\u0026hellip; For example you can extract data from PDF files and HTML pages. And lot more\u0026hellip;\nHash partitions in SQL Server\nIn SQL Server, when talking about table partitions, basically, SQL Server doesn’t directly support hash partitions. It has an own logically built function using persisted computed columns for distributing data across horizontal partitions called a Hash partition.\nASF 012: Rob Sewell interview\nDon\u0026rsquo;t forget to hear to this great interview we did with Rob!\nCheers,\nDamian\n","date":"2018-07-24T00:49:01Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/07/last-week-reading-2018-07-22/","title":"Last week reading (2018-07-22)"},{"content":" Introduction Rob is a SQL Server DBA with a passion for Powershell, Azure, Automation \u0026amp; SQL. He is an MVP \u0026amp; an officer for the PASS PowerShell VG \u0026amp; has spoken at, organised \u0026amp; volunteered at many events. He is a member of the committee that organises SQL Saturday Exeter, PSDayUK \u0026amp; also the European PowerShell Conference. He is a proud supporter of the SQL \u0026amp; PowerShell communities.\nHe relishes sharing \u0026amp; learning \u0026amp; can be found doing both via Twitter \u0026amp; his blog. He spends most of his time looking at a screen \u0026amp; loves to solve problems. He knows that looking at a screen so much is bad for him because his wife tells him so. Thus, you can find him on the cricket field in the summer or flying a drone in the winter.\nHe has a fabulous beard!\nThis talk has taken place during SQL Bits conference in London, on 22nd February 2018 (Thursday).\nInterviewers: Kamil Nowiński \u0026amp; Hubert Kobierzewski [T]\nWhat is his recipe for having a good speech? Where is a passion in his life? Check out which kind of toys Rob doesn\u0026rsquo;t take to abroad and why. Check that and also, why we had a lot of background noises during this chat (apologise for that).\nTranscript Kamil Nowinski: Hello, our today’s guest has the most recognisable and the longest beard among SQL family members, he is a PowerShell SQL automation guy, holder of MVP, blogger, speaker, PowerShell virtual group leader, DBA tools evangelist and one of the contributors of the tool, and probably it’s only a tip of the iceberg [laughter], probably I forgot about a lot of things\u0026hellip;\nRob Sewell: There’s plenty of things, there are plenty of things.\nKN: So, ladies and gentlemen, our guest is\u0026hellip;\nRS: Rob Sewell. Nice to meet you, hello.\nHubert Kobierzewski: Hello, Rob.\nKN: Nice to meet you.\nRS: And\u0026hellip; so that everybody knows that the background noise is because we’re sat in SQLBits conference.\nKN: Yes, absolutely.\nHK: 10 years!\nRS: 10 years, SQLBits, 10 years.\nKN: Yeah, exactly, very round number\u0026hellip; No, sorry, the round number is 16 [laughter].\n[everybody laughs]\nHK: So it was two years ago, that was the round number.\nKN: Rob – thank you for accepting my invitation to this podcast. And the first question is – where do you live?\nRS: I live in the South West of the UK. In the middle of nowhere, in a tiny little hamlet, I live next door to my dad on one side and my step-gran on the other side, it’s a beautiful place.\nKN: I saw some pictures\u0026hellip;\nRS: Away from the world.\nKN: Yeah\u0026hellip; Are you happy with travelling a lot? From there, you know?\nRS: What do you think? [laughter]\nKN: Yeah, probably I know the answer.\nRS: Considering the amount of\u0026hellip; I love travelling. So one of beauties of where I live is that I’m quite close to the M5, so I can very quickly get anywhere around if I need to\u0026hellip; Not quickly but I can get anywhere. And yeah, I’m lucky enough that my job and my passion means that last year I visited 17 countries.\nKN: Wow. That’s a big number.\nRS: Some for work, some for presentations\u0026hellip; So yeah, I literally flew all the way around the world the wrong way.\nKN: Singapore\u0026hellip;\nRS: Yeah. I flew to Singapore for PowerShell Conference Asia and then went from there via Japan to Seattle for PASS Summit and then flew back to London all in the space of two weeks. So that’s interesting. I wouldn’t recommend that.\nKN: No?\nRS: The jet lag is quite hard. The journey from Singapore to Seattle – you leave at 6 o’clock in the morning, you travel for 18 hours or something, and you arrive at 10 o’clock in the morning.\nKN: Yeah. I felt exactly similar when I was going to Seattle, last year. From London, so I spent 9 hours on the flight and it still was only\u0026hellip; I lost 3 hours, something like this.\nRS: Yes, yes. It really makes your body go “what’s going on?”.\nHK: OK, so – what do you do for the living? It’s, like, in general terms?\nRS: In general terms\u0026hellip; Very good question. I help organisations to do things that are automated, to make things go quicker and easier. So that’s real general. Would you like more specifics?\nHK: Not necessarily. But at some stage you have received an MVP title.\nRS: I have, yes.\nHK: So what did that mean to you, when you got it?\nRS: I found out that I got the MVP award whilst I was playing league cricket. So when you play cricket, you, one side, my side at that moment were batting, so we were not actually in the field playing the game, so I was waiting, picked up my phone, and just saw an e-mail alert come in\u0026hellip; Oh, wow! And it literally took my breath away, I was so delighted, I was so pleased.\nHK: Did you win that game?\nRS: We did win the game, yes, we did win the game. Yes.\nKN: On your website, I saw that sometimes Rob was also recognised a Microsoft Cloud and Data Centre MVP this year.\nRS: This year, yes.\nKN: It was last year, I think?\nRS: Ah\u0026hellip; Yeah. When I wrote it, it was this year.\nKN: Yeah. [laughter]\nHK: There are lot of people who are, like, looking at what you publish, what you present, what hints would you give to those, who are DBA wannabes, they start on the job market or they are kind of finishing their studies\u0026hellip; Are there any, like, three major hints that you would give them for their future career? What they should focus on?\nRS: Understand that you will never stop learning. Just because you are finishing your studies or you are doing your exams, that’s never ever gonna be the end of it. Even when your beard in grey like mine, you are gonna be needing to research and find out and learn about new things and new concepts. That’s number one cause that’s really important to know that if you’re gonna choose that career, this is the thing it’s gonna be.\nHK: Every day is a school day, yeah?\nRS: Every day is a school day. Never ever stop learning things. One of my mantras: never stop learning. And secondly, I would say, don’t aim on being a DBA. Aim on doing something that you love to do, you really enjoy doing, and that you do with a passion. Cause that way you’re gonna do something that’s much more fun. And it’s OK if you do\u0026hellip; I was speaking with somebody in our session today. She started as an application developer and then moved into doing some sort of BI things, and then actually found she enjoyed hearing about the DBAs and chose to find a job where that was gonna be part of her new job, and now she’s moving towards being a DBA. So it wasn’t really that she chose one thing. She chose to be an application developer, but gradually she moves forward, she’s gonna be for something else. And it’s kinda the same for myself, because I didn’t choose to be a production DBA. This is my third career. I’ve done other things in the past. And I just\u0026hellip; I was the guy who was good at computers so I enjoyed being around computers, and I moved forward, and I’ve got an opportunity to be a DBA, and I’ve found that I really loved it. And that’s how I started. And I was a production DBA for a long time. And now, I’ve moved really\u0026hellip; my work is not necessarily production DBA anymore. So it’s moved forward. So I don’t think you’re ever gonna sit still and be one particular thing. If you like that, it’s cool, and if you don’t like that, then that’s OK. Find a different thing that catches you. So that’s two things.\nThird thing, I think if you’re gonna be a production DBA, is to\u0026hellip; Don’t expect to be a top dog straight away. Understand what you’re gonna do. SQL Server’s really, really wide. So broad, so much stuff you can know\u0026hellip; So make sure that you can at least understand the concept of a lot of it, and then find the thing that catches your passion and your thing and then go deep into that. So\u0026hellip; I don’t really\u0026hellip; I understand the SSAS and cubes and that sort of thing, I can have a conversation, I can help that, but I really don’t know it very deep. Whereas some other parts of SQL Server, I do know much deeper. And I think as long as you accept that you know where the gaps are. Bonus tip.\nKN: If you want to be good at something, you should be focused only at one specific something, yeah.\nRS: So that was three. I will give you a bonus tip. Bonus tip is, people will give you a lot more respect when you say “I don’t know”. So if you don’t know the answer to something, say “I don’t know”.\nKN: Yeah, many people are just afraid of giving that answer, yeah?\nRS: Absolutely. They either feel that “Oh, I should know that”, you know, I think that inside sometimes “I should know that but I don’t”. And\u0026hellip; But also sometimes they go the other way and they try and bluff, and sort of waffle and talk and people don’t respect you if you do that\u0026hellip;\nKN: Yeah, absolutely.\nRS: So if you don’t know, say “I don’t know”.\nKN: So, you are giving a lot of speeches across the world, yes?\nRS: Yes.\nKN: And\u0026hellip; So, how do you prepare yourself for a speech?\nRS: So it goes back a long way, back to beginning\u0026hellip; So generally, what happens with me, is I will find a “thing”, whatever that “thing” particularly is at that moment in time. And I’ll normally write some blog posts about it as I go through solving a problem or working out what’s going on. And then there’s blog posts coming and I think: “you know what, actually this could make a nice presentation”. So then I start giving some presentation, thoughts in my head and I start making sure I got some demos about what’s going on. And then I might build some slides, and get an abstract together, and send the abstract off to people. And then if I’m lucky enough to get selected, then I will go back to my demos, I go through it, make sure that everything works as I expected, I’d write pester test to make sure my presentation is gonna work as I expected to do, because I blog about it, I do it, because it works for me really, really well. And then I’ll build my slide deck, and then I will practice. And practice, and I will practice, and then I will practice. And I will practice speaking out loud. My dog, my wife, my cat, the horse – they all heard these presentations as I’ve gone through them. And I will sit through\u0026hellip;\nKN: They must. [laughter]\nRS: I just go through, and go through, and go through, and go through\u0026hellip; And you are practicing to get used to the sound of your own voice, to make sure that you can comfortably say what’s going on. You practice without having the slides so that you can just keep on talking. And you practice also so you get a good understanding of how the talk’s gonna go, what the timings need to be. And no matter how much you practice the timings, the timing is never going to be right. And once you have done some presentations and you’ve spent some time doing it, you will be able to learn how you can vary that timing, manage it, speed up or slow down or whatever. But you only gonna do that if you can practice, and it’s really, really comfortable.\nKN: So you’re working a lot, I think, yeah?\nRS: Yeah.\nKN: You are engaged in many areas.\nRS: Yes.\nKN: So how about your work-life balance? You have something like this? Can you manage that?\nRS: So\u0026hellip; Do you know what my wife says?\nHK: There is no such thing like work-life balance?\nRS: What my wife says, she says “you are really lucky”. I say “What do you mean?”. “So your work is your hobby, is your job, is your passion, is your fun, is your social time, is your career, and those things can sort of merge into one another”. There was a day a few weeks ago, when I worked at home. And I worked at home, and I started at 6:30 in the morning, and I finished at 10:30 at night. And she said, “what have you been doing?”. Not in a bad way, just like “what was it that you’ve been doing all that time?\u0026quot;. And we were sort of talking about it. And I realised that some of it was work for clients and some work was writing a blog, and some of it was doing organisation stuff for the PowerShell Conference in Europe. And some of it was practicing presentations, and some of it was answering e-mails for other clients. And it all sort of happened in one big block, and you couldn’t really see which bits were work, which bits were clients, and which bits were community, which bits were presentations. It all kind of blurred into one. And I’m lucky because I have a very understanding wife, and also because the kids are growing up, so it’s just the two of us and the dog, and the cat and two horses. So, you know, I’m lucky.\nHK: Probably\u0026hellip; For sure we’ve just mentioned that, but how important is passion for you?\nRS: Passion is what you get with me. The things you hear me talk about, and that I blog about, are things I am passionate about, because I love what I do, I’m so lucky. Outside of IT stuff, I’m passionate about cricket. I play cricket for a league side, I like to watch cricket and I like to fly my drone.\nKN: Aaaah! That’s\u0026hellip; You won’t believe that, but this is exactly my next question [laughter]. So, my next question is: “Do you like taking photos?” And the extension of this question is: “How often do you use a drone for this?”\nRS: So\u0026hellip; Do I like taking photos – I love taking photos. I absolutely adore taking photos. One of the reasons I take photos is because my wife is not able to travel so much, and she likes to experience all of these places I go through all the photos I take as I go along. So I love being able to share all those with her, it’s really good fun. And yeah, I’ve got a drone. I’ve had drones for about 5 years now. I absolutely love it. It’s a real escape for me, because when I’m using a drone, I’m not sat at a desk with a computer in front of me, I’m out and about walking and doing stuff. I gotta go somewhere particular, I gotta check if it is a place where I’m allowed to fly the drone. I’ve gotta go and find something interesting to take a video of. And I go off and I take videos, and I take photos.\nKN: I saw your photos from the drone. Yes, so it was amazing quality of the photos.\nRS: Really, really good, really excellent.\nKN: Yeah. I was so surprised.\nHK: Do you take your drone with you when you travel?\nRS: I haven’t taken my drone abroad. I need to think about\u0026hellip; If it’s going in an aircraft, then I need to think about all the batteries and where they would be. I always worry, I take an awful lot of electrical equipment, as you imagine, so customs are always fun anyway. But so far I’ve only ever done it in the UK. I’d love to take it abroad but haven’t managed it quite yet. Cause there’s an awful lot of rules and regulations, so within the UK you are only allowed to fly up to 400 feet, you have to be within eyesight. There are certain places where you are not allowed to fly and\u0026hellip; I’m not sure which authority name is precisely, but they provided an app with\u0026hellip; That gives you the map that shows you where you can and can’t fly. So I’m very, very careful to follow all of those things and I haven’t really researched any other country yet. But undoubtedly, at some point I will take it abroad, yes.\nKN: What kind of role do you play at the SQLBits Conference? Apart of, you know, being a speaker. Because you have finished two full days preconfs and another two regular sessions are ahead of you, so\u0026hellip; But also, you are very deeply engaged in organisation of SQLBits.\nRS: So in the organisation of SQLBits I have absolutely zero input at all. Nothing. I do nothing, I’m not a member of the committee, no part of the organisation at all.\nKN: Really? Oh.\nRS: Really, yeah.\nSo, this year, I did play a small part in some of the speaker selections, and up until then, my role has been as a helper, as a volunteer. So because I’m in the South West where Jonathan and Annette are, they are both members of the (SQLBits) committee, we run the user group together, did SQL Saturday Exeter together.\nKN: Ah, so it’s easier, working together.\nRS: Yeah, we know each other and we get on really, really well together. So I love helping people and I’m not afraid to talk to people. And\u0026hellip;\nKN: Yeah, we know that. [laughter]\nRS: So\u0026hellip; I really enjoy doing that part of it. So I’ve been the conference helper, the runner, room monitor, helping to make sure that it happens, in fact that’s what I’m doing for the next two days. I’ve got two sessions as well, but I’m gonna be a conference helper, running around, making sure that everything is happening smoothly, because you cannot imagine the amount of work that goes on to put on a conference the size of SQLBits. It’s done by volunteers in their own time and that’s before it happens. And then when the thing is actually going on, you need to make sure that the rooms are in the right place and the speakers arrive in right place, and they are starting and finishing at the right times and the audio is in the right place and that coffee is coming. And all of these things. And you’ve got 2000 people here, and they need to know where things are, and they have problems and issues they need to get resolved in certain ways, things break. And all of that is gotta come through and be organised in some way. So I play a tiny part in that.\nKN: Yeah\u0026hellip; Actually, we can imagine that, because we are organising SQLDay in Wrocław. Of course it’s not that size of the conference, yeah, but still.\nRS: Not only that, both of you got orange shirts on, you’re both volunteers here as well, doing your part and helping.\nKN: Yeah, both sides of the story.\nKN: So you are blogging and tweeting a lot. Your last post is about 108 configurations.\nRS: Yes.\nKN: Today’s blog, today’s post. Tell me something about Pester and how it can help for DBAs?\nRS: So I blog and tweet a lot\u0026hellip; Yes, because I like to share what it is that I do. My point of view on that is that there is no point you guys going off on writing something if somebody else has already done it. Just like, I don’t go out and write something new when somebody else has already written it and I’m making use of it. And this is how we all learn and share and improve, and take ourselves forward. So Pester. I love Pester. Pester is one of my favourite things. I’m a big PowerShell person, I’ve always loved using PowerShell. And Pester enables us to\u0026hellip; is a unit testing framework for PowerShell. It’s got two sides to it. You can use it in test-driven development (TDD) to write your PowerShell code in a good way, make sure that you are unit testing it correctly, it’s doing what you expected it to do, as you are writing it. Good, proper code writing. You can also use it withing the integration test, which is what we do in DBA tools. So DBA tools before a command goes out and is released to the public, it has already run an integration test against actuall SQL Servers, and we’ve made sure that all of the things that it’s supposed to do or what it needs to do, by using Pester to validate that’s what’s happened. And today, we announced DBA checks. And DBA checks enables anybody to run over\u0026hellip; The are 80 tests, so there’s 109 configurations for those tests that enable SQL Server DBA to be able to validate that their environment is as they would expect it. Pester gives a nice good output. Green is “pass”, red is “fail”, so I like to say, green is good, red is bad. And we can also export it out into a JSON or any xml for use in CI processes, convert it to html. But best of all, you can prepare BI in front of it and make pretty dashboards.\nKN: Yeah, that’s great. So in terms of Pester, so, does it mean that it’s not only for DBAs? If you use PowerShell, you can use Pester as a tester tool.\nRS: Absolutely. So if you write your PowerShell code, you should use Pester to validate it with the unit test integration test and if you can write, if you can run a PowerShell command and get an output, you can write a Pester test to check if it’s the output that you want or not. And run it repeatedly and reliably, automated overnight and get results.\nKN: Aha, great, because so far I thought that it’s mainly for DBAs, but it’s for developers as well.\nRS: Anybody. Azure guys, admin guys, sysadmins, Active Directory, Exchange\u0026hellip; There is no limit to where Pester is gonna stop. If you can connect to something with the PowerShell, you can use Pester to test it.\nHK: We said the word “PowerShell” a lot during this interview. So the question is, what was that inflection point in your career that turned you to PowerShell? Was it any session or you’ve seen the speaker, or\u0026hellip;?\nRS: So, it was literally that validating my agent jobs had successfully completed before I even knew Pester, before Pester was even around. That was the thing enabling me to go on 100, 120 instances, and make sure that every single job works\u0026hellip; And what I used PowerShell to do is to create and Excel sheet and turn a cell green for good and red for bad. So that’s where it really started and I went “this is the thing that works”. And many years later, I’m still doing the same but I’m doing it with Pester. So it’s really quite freaky, it’s quite funny.\nHK: Yeah, good point, valid point. OK. What source code control environment do you recommend for continuous delivery of data bases?\nRS: GitHub, or Git. Git is the sort of control you should use.\nKN: Why do you prefer Git?\nHK: Any reasons?\nRS: What is your other option?\nHK: Team Services.\nKN: TFS?\nRS: TFS, VSTS, they all work, I just like using Git.\nHK: OK. So, what do you think about the role of advanced analytics in the role of DBA?\nRS: What do you mean by advanced analytics?\nHK: Like analysing logs, predicting situations, predicting problems\u0026hellip;\nRS: I’d say it’s the way the world is going. This is what we are going to do. There is no point trying to use human beings to analyse hundreds and thousands of files and text or whatever, because computers can do that so much better. That’s where we’re gonna be, we need to accept that. We were talking earlier about never stopping learning, always keeping moving forward. This is the way that we’re gonna start moving forward, so we have to accept that.\nKN: Do the DBAs should worry about it? And try to find a new job?\nRS: No. DBAs shouldn’t worry about finding a new job, DBAs should just adapt to survive, to do what they need to do.\nKN: At the end of our conversation, tell us, where we can find you on the Internet, you know?\nRS: Finding me on the Internet is really easy. Search Google for “SQL DBA with a beard”. [laughing]\nIf you wanna find me on Twitter, you’re gonna have to do “SQL DBA with beard”. So there’s no “a” there.\nKN: We’ll add a link to that. And to your blog as well.\nRS: Thank you very much.\nKN: Thank you very much, Rob.\nHK: Thank you very much.\nRS: Thank you.\nUseful links: Rob\u0026rsquo;s twitter: @sqldbawithbeard\nRob\u0026rsquo;s blog: sqldbawithabeard.com\nPowerShell community: DBAtools.io\nPowerShell Virtual Group: PASS Website\nConferences: SQLBits, PSDayUK\n","date":"2018-07-18T06:40:30Z","image":"/uploads/2018/07/IMG_9273_ASF012_SQLBits_with_Rob_Sewell.jpg","permalink":"/2018/07/asf-012-rob-sewell-interview/","title":"ASF 012: Rob Sewell interview"},{"content":"Structured Streaming with Databricks into Power BI \u0026amp; Cosmos DB\nGiuliano Rapoz (Microsoft) is showing the concept of Structured Streaming with all these puzzles.\nLightning fast query performance with Azure SQL Data Warehouse\nThe post shows how the shuffling speed has improved using SQL DW instant data movement mode.\nAzure sets new performance benchmarks with SQL Data Warehouse\nHave a look at 30TB TPC-H Benchmark results for Azure SQL DW and Amazon Redshift.\nNew Azure #CosmosDB Explorer now in public preview\nIf you work with Cosmos DB this tool might me useful.\nPredicate Pushdown and why should I care?\nUnderstand better (or ever) the differences between Number of Rows Read, Estimated Number of Rows to be Read and Estimated Number of Rows.\n","date":"2018-07-15T22:53:57Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/07/last-week-reading-2018-07-15/","title":"Last week reading (2018-07-15)"},{"content":"Link to this (past) event: http://www.sqlsaturday.com/730/\nPictures [srizonfbalbum id=13]\n","date":"2018-07-15T19:14:43Z","image":"/uploads/2018/07/sqlsat-730-manchester.jpg","permalink":"/2018/07/sql-saturday-730-manchester-pictures/","title":"SQL Saturday #730 Manchester (pictures)"},{"content":"Hi all,\nThe next episode of ASF podcast is almost ready.\nDo you know who is the next guest?\n","date":"2018-07-13T00:07:56Z","image":"/uploads/2018/07/ASF012-video-invitation.png","permalink":"/2018/07/asf-012-video-invitation/","title":"ASF 012 video invitation"},{"content":"Welcome to the next post regarding Power BI.\nThis time, we will know Power BI from the more pleasurable side - that is, it will be practically and conveniently. Namely, we will deal with downloading data from Excel, the internet and of course SQL.\nPower BI loves to load data. In that case, let\u0026rsquo;s give it some nutrients in the form of data that must be downloaded and loaded. In this post at the beginning I will briefly describe where you can save data and how to download and load data to Power BI.\nThat was a theory. Let’s go to practise knowledge.\nHow to get data from Excel to Power BI In these part of this post I show you how we can do this from local desktop. It’s so easy and it’s so cool.\nFirstly, you must prepare your workbook for Power BI. That means all your cells which contains the data must be the same colour, without total rows. Data and columns should be the same types. If your data are in outline or matrix format or pivot tables - they must be formatted as a table. And the most important things - Workbooks must be saved as .xlsx or xlsm file type and be under 1GB.\nThe last - good way is get your table name it will be simply recognize it in Power BI.\nOK, let’s start get data to Power BI from Excel.\nWhen you open Power BI desktop click Get Data and chose Excel (as you already know, you can find it on the left side ribbon).\nBecause you chose Excel so the previous menu navigates you to your Excel workbooks. Now, it’s the place where you should click your file then you will see following window.\nFor this example I used my excel file. I find them in Wikipedia.\nAt the moment you can be little confused what should be clicked – Edit or Load? Your choice depends on what you want to do with this data. If you choice Load then Power BI loads your data to data model. But if you choice Edit Power BI opens queries editor. Here you can prepare data, modelling columns, delete column, rows and (this is the one of the best part of Power BI) you can load these clean data into to data model.\nJust like you see - get data from Excel to Power BI is so easy. In the next step in this post I’m explaining how to get data from a website.\nHow to get data from website to Power BI Get data to Power BI Desktop from website is not difficult. First we need website where we can practice this case. I chose Wikipedia - we will be search over there a population in UK. I have found this page for you here: https://en.wikipedia.org/wiki/Demography_of_the_United_Kingdom\nIf you have opened the Power BI Desktop, select Get Data =\u0026gt; **Web.**Now copy and paste URL mentioned above and click OK.\nWhen you do this you must connect to this website.\nWhen Power BI connects to website you will see Navigator window.\nOn the left hand side you can select different part of this page. You can see all table available on this site. When you click one of the check box on the right site it will be open the contents of the selected table.\nFirst you will see then we look at the table view we can see correct data. Also, on the right you can see two buttons: Table View and Web View.\nWeb View Here you can see how this table looks in the Internet. For example.If you select table “Age structure for men and women in 2011[33]” you will see how this table looks on the website.\nTable view Just like above you can choice Load or Edit depending on what you want to do. (See above).\nHow to get data from SQL Server to Power BI Now, I want to show you how to get data from SQL to Power BI. A beginning is as similar as getting data form Excel to Power BI.\nFirstly, of course you know that, we must click Get data and then SQL Server:\nIn order to connect to SQL Server’s source you must enter the SQL Server name and the Database name.\nAs you can see in “Server” I entered name of my local server. In second window I put name of my Database - in this case it’s AdventureWorks2012.\nWhen you finish this part and click “ok” and you will see following window.\nOn the left you will see Display Options. There you can see a list of tables, views and functions listed for that database. Click required object and load them to Power BI. If you click one of them - there is a preview how the table looks like, on the right. How many columns they have, the name of its columns and how many rows it has.\nFor this example: I choose Person.PersonPhone and on the right I can see that this table has 6 columns, 22 rows etc\u0026hellip;\nOnce you loaded the data, you can edit the model in Power Query or start build your report by dragging fields into the design area. Use Power BI visualization, add slicers and whatever you want. It’s your decision – have fun.\nSo, now you will know which type of source you can choice and how to put data from Excel, Website and SQL Server to Power BI desktop.\nIn next post I will show you how to edit all those data in Power Query editor.\n","date":"2018-07-11T23:25:56Z","image":"/uploads/2018/07/aqua-3445987_1280.jpg","permalink":"/2018/07/getting-the-data-to-power-bi/","title":"Getting the Data to Power BI"},{"content":"Good day SQL folks!\nMonday morning is always tough to start working on serious tasks. I think it is a perfect time to read something interesting before you really go into tons of tasks. Now let me give you some things to read for the rest of the day.\nThis post was scheduled to be delivered on Monday - as usual. And Monday won with me just by sending too many tasks immediately in the morning. Now I can say that \u0026ldquo;Monday morning is always tough to start working on serious tasks. I think it is a perfect time to read something interesting before you really go into tons of tasks. Now let me give you some things to read for the rest of the day\u0026rdquo; .\nNow (Tuesday early morning) let me give you some things to read for the rest of the day!\nPredicate Pushdown and why should I care?\nPedro Lopes (check the interview with Pedro here) explains the concept of a predicate pushdown and its inplications during the optimization process.\nWhat is the FCB_REPLICA_SYNC spinlock?\nPaul Randal explains this wait type. You should not see much of it unless you use a database snapshot. Then you will see trillions of them\nTEMPDB – Files and Trace Flags and Updates, Oh My!\nPam Lahoud\u0026rsquo;s blog post about our favourite topic - the tempdb. You can read about some recent changes that have been introduced to this database.\nDeploy Azure SQL Managed Instance network environment using ARM\nAzure SQL Managed Instance is fully managed SQL Server instance hosted in Azure cloud. You would need to prepare Azure environment where SQL Managed Instance will be hosted before you create your Managed Instances. In this post you will see how to deploy configured Azure environment using ARM template\nModeling document data for NoSQL databases\nShowing how different can be modelling documents for NoSQL databases from what you have known from the relational world\nASF 011: Grant Fritchey interview\nDon\u0026rsquo;t forget to hear to this great interview we did with Grant!\nCheers,\nDamian\n","date":"2018-07-10T03:58:41Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/07/last-week-reading-2018-07-08/","title":"Last week reading (2018-07-08)"},{"content":"Apart of standard portion of news which we have for you every week - there is another extra info from us. Firstly, we\u0026rsquo;re proudly to announce that Damian Widera has been honoured 10th time in a row with MVP award! MVP is a very prestigious title being awarded by Microsoft to people who have an excellent technical knowledge and significant impact into the community. Most of our guests are MVP, so also we\u0026rsquo;re very often touching this topic during the podcasts. Congratulations Damian!\nSecond thing is about my coming activities. On Wednesday I will be presenting my session about Azure SQL DataWarehouse at Southampton Data Platform and Cloud Group Meeting [T]. Furthermore, 9th of July (Monday) I will show this session again, this time at local group in Guildford at SQLSurrey PASS User Group [T]. I\u0026rsquo;m very excited about this but bit nervous at the same time as it\u0026rsquo;s my first time when presenting in England. Not my first time to present a session in English though, but first time in England to native people.\nOk, it\u0026rsquo;s time for usual news. Have a good week!\nProcess more files than ever and use Parquet with Azure Data Lake Analytics\nMichael Rys reveals how ADLA is now more efficient processing hundreds of thousands files and supporting native Parquet format.\nReceive SQL Server Alerts through Slack\nCheck this trick out - how to send alerts from SQL Server to Slack.\nEvent trigger based data integration with Azure Data Factory\nNew feature in Azure Data Factory (ADF) pipelines, where now you can trigger an event based on files changes (appeared or deleted) in your blob storage.\nAzure Data Architecture Guide – Blog #1: Introduction\nAmount and capabilities of technology about the (big) data are continuously growing. This Guide prepared by Microsoft AzureCAT (Customer Advisory Team) explains when which particular technology should be used in which scenarios.\nAzure SQL DW Performance: CTAS/Partition Switching VS UPDATE/DELETE\nUPDATE and DELETE are not very performant operations in CCI. What is the CTAS then and why it\u0026rsquo;s much faster than UPDATE? Check this in a practical example.\nThe emerging big data architectural pattern\nLambda architecture is a popular pattern in building Big Data pipelines where we distinguish speed and batch layers. What kind of problems it introduced and which it solves?\nAzure Elastic Database jobs is now in public preview\nThis is cool - you can have SQL jobs in Azure. No, not in neither VM nor Azure SQL Managed Instance. Just on Azure SQL Server level.\n","date":"2018-07-01T20:36:16Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/07/last-week-reading-2018-07-01/","title":"Last week reading (2018-07-01)"},{"content":" Introduction Grant Fritchey is a Microsoft SQL Server MVP with over 20 years of experience in IT, including time spent in support and development. He has worked with SQL Server since 6.0, back in 1995. He has developed in Visual Basic, Visual Basic .NET, C# and Java. He is the author of the books SQL Server Execution Plans (Simple-Talk), SQL Server Query Performance Tuning (Apress) and co-author “Expert Performance Indexing”, “SQL Server MVP Deep Dives 2”, “Beginning SQL Server 2012 Administration” and “Pro SQL Server 2012 Practices”.\nSince 2009 he has been awarded as a Microsoft SQL Server MVP. In 2014 I was awarded as a Dunn \u0026amp; Bradstreet MVP. In 2011 I received the Tech10 Award in Rhode Island and joined Red Gate Software as a Product Evangelist in January.\nGrant volunteers at PASS, currently serving on the PASS Board of Directors as the President.\nThis talk has taken place during PASS Summit in Seattle, WA, on 3rd November 2017 (Friday).\nDo you wanna know why Grant is so-called Scary DBA? How is he sharing his working time between Redgate and PASS and who has given him 6-weeks sabbatical? What is his approach to teaching, speaking and working? What does mean work-life balance for him?\nCheck this out right now.\nTranscript Kamil Nowinski: This is a podcast for SQL family members from SQL family members. Especially from the well-known SQL family members, like you.\nGrant Fritchey: Excellent.\nKN: We are going to talk not only about the technical stuff but we will focus more on the community, volunteers, organizations, etc.\nGF: Sure, not a problem. Happy to help.\nKN: Could you introduce yourself?\nGF: My name’s Grant Fritchey. I work for Redgate Software. I’m a Product Evangelist. I’m also a volunteer for PASS organization. Currently I’m serving on the PASS Board. I’m the Executive Vice President for Finance and Governance but starting in January I will be the PASS President.\nKN: Congratulations!\nGF: Thank you.\nKN: Were you announced like an hour ago?\nGF: I’ve been announced a little while back but yeah, we’ve made a big deal about it.\nKN: Good for you!\nGF: Thanks. It’s going to be a bunch of work but we’ll still have fun.\nDamian Widera: But I think it is not only good for Grant but also it is good for the whole organization and also for us because Grant is a very energetic person. You are the right person in the right place.\nGF: Let’s hope. But I’ve got the real focus on the global goal family. We met over in Poland. And I like coming over there. I will go there as soon as I can. I am really into the whole idea that this is a global organization and global SQL family. The PASS organization needs to do more to support global growth.\nKN: Could you explain a little bit in details what are you doing? Let’s start with the PASS.\nGF: Currently I’m the Executive Vice President so I do the finances. That means I work with all the other portfolios on their budgeting and figuring out where we’re going to spend the money and how best to spend the money and how much money we’re going to make and where do we take it then from there. The big part of this job is managing the cash – looking at spreadsheets. God knows, I didn’t realize that I was volunteering to look at spreadsheets. As well as that, I also help managing the meetings. We set the agendas and then make sure that we stay on target and get the meetings done – clean and quick – so it’s very little noise and just information so that we can meet the goals and get out the door and stop. I hate meetings. The less we can do, the better. So we manage those. That’s my current work I’ve been doing at PASS. But I’ll become the President and I don’t know what that entails yet, really.\nKN: You’ll realize very soon.\nGF: Yeah, no kidding.\nKN: It sounds like very serious bunch of duties.\nGF: You know – it’s work. You put in several hours a week at the minimum. Sometimes more. Like here, at the PASS Summit – it’s been pretty much nonstop. I do a lot of work here. We have meetings with vendors, Microsoft, Microsoft execs. also meetings with each other and the community. We’ve just finished an hour of question-and-answer meeting with the community trying to engage so that we’ll know what people want and frankly – where we are not getting the job done correctly so that we can constantly improve.\nKN: How can you split your time between PASS and Redgate?\nGF: I’m very fortunate with my job. Redgate Software is a great, great organization. They are active supporters of the community. They support the SQL Saturday events and other events all over the world. They believe in the concept of the community so they supported me running for the Board. I’m given the dispensation to spend a little bit of my work here, occasionally, on the community stuff. I’m just very thankful for that. I’m lucky.\nKN: Going forward to the Redgate now. What are your main duties there?\nGF: Oh, I stand around at parties… [laugh]. My main thing at Redgate is to learn how this software works. I have to learn as much as I can about the tools and then I have to learn how to apply them in the businesses. Then I basically come back around and teach you guys how to do the same thing. That’s the big part of what we do – I get an understanding and then I share that understanding, either through webinars, videos or articles. I also occasionally do consulting on the side so I travel to some places, talk to the DBAs and work with them to fix their DevOps solution.\nKN: From that perspective it is very good to have a direct connection to the people.\nGF: Yes, you want to make sure that you’re attempting not only to solve the problem that they need but also that you understand what problems they have so that you are building the tools that they need. I do that. And then I work with our dev teams because, as a DBA by experience, I bring a perspective that they don’t have and I can work with them to help set the agenda on the projects.\nKN: How is your work-life balance?\nGF: What is life-thing you talk about? [laugh] It’s pretty good. It really is. I’m really lucky because in Redgate they take pretty good care of their employees. Every 5 years they give you 6-weeks sabbatical. And so I just recently took mine for the first time, even though I’ve been there for 6 years – I delayed it for a year. So I had basically 6 weeks to spend at home with my wife. For three weeks we went to Germany, because I’m trying to learn how to speak German.\nKN: Oh really? We saw your pictures from there on Facebook.\nGF: We had a really good time. She got to see me for 6 weeks so the fact that I am away from home…\nKN: That’s for the balance.\nGF: That’s it – 6 weeks every 5 years! No, I’m kidding. They take really good care of us. My travel time is about 50% so I’m about half the time on the road. But the other half I work from home so I’m able to set my own schedule and get things done when I need to: go camping with the kids, go work out – I need to do a bit more of that, obviously. Step away from the dinner table. I get to do a lot of that kind of stuff. It works out fairly well since it’s only half the time on the road.\nKN: Tell us and our listeners how you start your adventure with SQL Server.\nGF: I’ve been working with it for 25 years or so. I started out initially as a developer and I was doing a lot of VB back then and I moved onto C# and some Java. Initially SQL Server was just this thing that I used to work on the back-end but what I found out was that I spent more time messing with the database, making sure we had right indexes, changing the database design. I spent more and more time on the database. So at one point in one of my jobs we didn’t have a DBA – I found that later why. And I went to my boss’s office one day and said: “Look. This is going wrong.” And I kind of went off on a crazy rant – yelled and screamed. “This is the problem, this is the problem and this is the problem.” And he started writing all the problems down on the board. I kind of calmed down because I saw that he was listening to me. And he turned and said: “Ok, so which one of these you think you are to address first?” And I thought to myself: “What?” And suddenly I was a DBA. It was kind of shocking! That was how I started as a full-time DBA. It was about 20 years ago.\nKN: Yes, a long time ago. Damian, do you have any more questions in that area?\nDW: I will ask only one – what is your favorite tool? What do you use the most?\nGF: Redgate SQL Prompt – of course! But apart from Redgate – what is my favorite tool? Let me think about it for a second. Funny enough, it’s not a database tool. The one tool that gets me through my day – because I have to record videos and all these other stuff – I really appreciate Techsmith Camtasia and Snagit. They are the two tools that I use all the time because they let me do the screen captures in a really highly-controlled fashion and also they allow me to record videos. I use those a lot! But SQL Server tools – I can’t pick a competitor so I have to think about this. I know! Cios! Cios has got some really interesting stuff going on. I’ve been seeing what they are doing with it and it’s pretty amazing. They’ve put machine learning into some of the monitoring products that they are doing on Azure. It’s really amazing.\nDW: I would like to ask you about what do you do for community or how do you share knowledge? I know your blog and I would like to ask you why do you call yourself “Scary DBA”? What’s so scary about you?\nGF: I don’t think I’m scary at all. The story is really simple. My previous employer… when I first started we only had a few DBAs there and frankly some of the DBAs that we had at that time, they translated over from the Oracle side. They weren’t always that great. A bunch of them learned stuff and we became a great DBA team but when we started it was little rocky. One day the developer came over to me and said: “I’ve got a problem. Can you help me?” And I sat down and walked him through the problem. He went back to his dev team and said: “Hey, I’ve got the solution to the problem. I’ve talked to a DBA.” And they were like: “Wait a minute! Which DBA did you talk to?” because there was this concern that some of the DBAs weren’t quite getting things right. And he said: “Oh, you know, I’ve talked to the scary one.” Then everybody in the room went: “Oh! Grant!” and then they started laughing and called me up and said: “You never guess what happened!” And they told me the story. The next week a name plate shows up on my desk – Scary DBA. So everyone would know who I was.\nDW: So now we know the story!\nKN: There are many volunteers working during the PASS Summit conference and the SQL Saturday. How could you encourage the people to be volunteers?\nGF: First, I would start off by going to your local groups wherever they are – I know you have a bunch at home. That is a good place to start and frankly, I would volunteer to speak. Because local groups are pretty much always looking for speakers. It’s a constant thing. So – volunteer to speak. Everybody can teach something, everybody knows something that the rest of us don’t or you have a better slam on something that the rest of us have. I would start there. Maybe help out with the local group – help organize the pizza one night, see if your company can sponsor or host the meeting. The more you do it, the more you’re going to recognize that it’s all about meeting people in networking. A lot of people are saying that networking is about looking for work. It’s not! Networking is about making contacts with people. If you make contacts with people, can it help you get your job? Sure, it can. But more importantly, it helps you to build up a knowledge base. You know, I don’t know everything. I can’t know everything. So I have some friends I know who are really good at SSIS, and I also knew some people who are doing analysis – services or power BI. If I have a problem, I can send off a quick question: “Hey, when you have 5 minutes, can you help me out with this?” It makes me a better employee because I’ve got a broader scope of knowledge available to me rather than just the stuff that’s inside my head or what I can google. I’m going to get better answers from friends and acquaintances who are experts at different things. The networking is a huge aspect. It’s just invaluable to your career to have a good network of individuals – fellow DBAs and developers, analysts – you won like a diverse board of people, you’ve built your own little board of directors that you can talk to and get help from. If some of them will be the consultants, they’ll say: “I’ll give you 3 minutes!” And then you’d have to pay for it! But you still get that 3 minutes. So it’s just the great way to go.\nKN: Yeah, but still so many consultants have an MVP title?\nGF: Sure, there are a bunch of them.\nKN: So I think that they should be obligated to answer your questions, at least!\nGF: Well, a friend of mine, Jeff Hiten, put it: “The MVP has a helping gen**e.” We can’t help it. If you come and ask us for help, we’ll help you. It’s just how it goes. I think it’s just a big part of how you make an MVP – you’ve got that gene built in your DNA.\nDW: “Helping gene” and of course the wish to share the knowledge because you can also learn by sharing the knowledge.\nGF: Oh, yeah. I learn more by doing the presentation because I have to structure everything, set up the examples. I have to test it and validate it. And then, when I am presenting it, I get asked questions. I don’t always know the answers to those questions so it becomes something else that I have to go in and look up. This is this nonstop knowledge. It’s wonderful.\nKN: It was proved by the scientists that you can learn the most if you are preparing yourself to the presentation and sharing the knowledge.\nGF: So scientists say it’s ok – awesome!\nDW: How do you prepare yourself to give the presentation or the talk?\nGF: The hard part is deciding what to talk about. Figuring out how to say something new about the topic or how to have a new wrinkle on a topic. You could just teach the same stuff over and over again, obviously, but you always want to add and change something. So coming with the topic is the hard part. But once I’ve got a topic, I start to drilling down on the data – I read articles on it, I write articles on it while I’m doing stuff. I will spend a lot of time with the database and just drill down on whatever it might be and figuring out the ways to break it or modify it or make the behavior change. Frankly, I call it playing. I go and play. Let’s say you want to look at batch mode processing in an execution plan, so how do you do that? You go in and set it up, right? Create a table or create a database. Add some data in there or slap a columnstore index on it. Run a query: “Did anything happen? Ok, what if I run a query differently, what happened? What if I change the data, what happen? If I force the plan to go parallel, what happened?” And just work your way through that and then once you’ve got the knowledge, then you just start to say: “Well, hang on! How did I have to set this up: 1, 2, 3, 4? What do I have to do to modify it: a, b, c, d?” Awesome! There is the presentation: 1, 2, 3, 4 – a, b, c, d. We’re done. Just be able to walk through and explain it to people. I do rehearsals. I will walk around my room, running through my slides and demos and talking to myself. My wife makes fun of me. I do practice everything. Because you don’t want to do anything for the first time standing in front of a hundred people. That’s no fun. Although I’ve done that and that’s how I know it’s no fun. That’s pretty much it. It’s really coming up with the idea and then play with the topic until you’ve got enough knowledge to put together your presentation.\nKN: On Twitter, as I correctly remember, you have more than 10,000 followers?\nGF: Yes, I just broke 10,000. It was like two month ago.\nKN: Do you feel like the public person?\nGF: No, I don’t. I try to put myself out there and be accessible to people. My goal is to be helpful. I’m not that smart but I’ve been around a long time. I know a few things and I feel like I can share those things. Frankly, I got a good feeling out of sharing. It’s kind of selfish really. It’s not so much that I’m some wonderful, nice guy. If somebody says: “Man, your script helped me!” That’s why I do it. “I read your blog and it gave me a piece of knowledge that fix something at my company!” That’s how I win. Those little moments where somebody says: “That helped.” It’s like: “Ta-da! I win.”\nKN: That’s your prize from your work.\nGF: Seriously, that’s my prize. And just come to find out that some companies will pay for you to do that.\nCheck audio version to work out why this picture is right here\u0026hellip; KN: True. I’ve asked about your Twitter and your followers as well, because I’ve seen many times that you share some articles that you’re reading. And it’s not about the IT. It’s a completely different area.\nGF: I do read a lot of stuff on IT but I also read a lot of stuff on how people think, how to better communicate or how to put together presentations. I read a lot of science articles on space because I’m just interested in that. I read tons of articles on history. I’m a big history nerd. And if I found a good one, I mark it. Then I share it later. During a week I just spit out two or three links to articles that I found interesting. Lots of them are about SQL Server but sometimes it’s how to write a better presentation.\nKN: Sometimes it’s good to read something else.\nGF: Yes, and it gives your brain a rest.\nKN: Yes, exactly.\nDW: I think, Grant, that your journey to Germany was pure historical because you’ve been in so many historical places.\nGF: Oh, yes. We hit a museum almost every day – my wife and I. I’m a hardcore history freak – walking around looking at the architecture, visiting certain locations. There were many great places. I don’t want to pick up just one place and say it is the best place. But the best specific location that I most enjoyed was going to the Battle of Nations in Leipzig. There is a big monument there which was just stunning. That was a special moment. But that’s a Napoleonic War thing for anybody who doesn’t know.\nDW: I have one question about the community. As you were explaining how you prepare to give at least a good presentation – however I think that you give perfect presentations – but how would you like to convince a typical person to prepare to be a speaker during a local meeting?\nGF: It’s easy. Let’s say you’re a DBA. And let’s say that you’ve got a development team who try to do something that’s gonna hurt your database, like they want you to put in all heap tables or something crazy like that. How do you convince them to not do this crazy thing? Well, you have to explain to them what’s going to happen when they do this and how it’s going to hurt performance, the various effects that it could have. That is basically the presentation. You are absolutely going to be teaching someone and putting across your point of view. If you do that really poorly, then they’re going to drop all the indexes on the database and everything is going to be a heap table and you’ll be living with that for a long time. So since we want to avoid this heap table problem, let’s learn how to give an effective presentation and learn how to talk through these points and attack them from a logical standpoint in such a way that I can convince someone that I’m right and that I’m going to be helpful to them. It’s not simply: “You’re wrong” but it’s rather saying: “Here’s what happens and here’s what we could do to make things better.” Just talking to strangers and walking through this process is a great way to train and practice. Then you come back to your business and you do it there. It’s the same when you want to convince your boss to give you a rise – that’s a presentation, right? If you are in a job interview – that’s also a presentation. You’re actually doing presentations a lot. You just don’t think about them that way. So it’s actually worth it to practice presentations. And a good way to do that is by going to your local group and doing the presentations. You don’t have to be trying to be MVP. But just for your own personal business and your own personal life, practicing your presentations is a great way to go. I would also say the same thing about blogging. It’s practice writing, and you have to write emails, business proposals, evaluations. So practice writing by writing a blog. Further, we want learning things and you’re learning them one step at a time and if you write them down in a blog, it makes it a lot easier to remember them later. Also – keeping a blog or something like that acts as a way for employment for you so that you can show businesses what you’ve learnt and what you’ve been doing. Any of these things that we are doing is not simply becoming an MVP or going on the road and speaking in 27 different countries. It’s just about being better in your own job.\nKN: Are you a perfectionist in your personal or professional life?\nGF: Am I a perfectionist? Good God, no! No, no, I’m not. I’m really not. I’m largely very compromised individual. I just do my best to get by. I couldn’t do a perfect thing no matter how long I tried.\nKN: But probably by most or at least by the all people from the community or the attendees of such conferences you are seen as a very good professional.\nGF: My approach is not to be a perfectionist. My approach is to try to do as much of the right things as possible and deal with the fact that you’re not always going to get it perfect. Would I want my database just to be purely designed and tuned to a fair-thee-well, and everything’s just shiny? Yeah! I’d love to get there, but I’m not going to. I’ve accepted that. I’m just going to go for “good enough”. I’m going to go for “this meets the business needs adequately”. It doesn’t have to be perfect. It just needs to be good enough. Like an execution plan – good enough!\nKN: But not perfect. There is nothing like a perfect thing. It’s good to hear that because we’ve already heard that from many people that the perfectionism is not the goal itself.\nGF: Yes. Honestly, a lot of people will look around and say: “I’m not going to do this because I can’t be as good as you”. It doesn’t matter. One, you probably will be better than me. But you want to be good enough. You don’t need to be perfect or great, just be good enough. People shouldn’t stop trying. They should keep going.\nKN: You can be a perfectionist in your own way, yes?\nGF: Yeah, of course.\nDW: This drives me to one more question about your blog where you put lots of articles with great quality. In my opinion at least most of them are really helpful. Imagine that there is a new person who would like to start blogging and this person says: “Well, there is Grant, there is Aaron Bertrand, Brent Ozar, Paul Randal and many other great authors, what can I write? I have maybe 1% of their knowledge. Should I start? How should I start?”\nGF: Oh, that’s an easy one. I can tell already the same thing. Ok, I Shouldn’t say that. If you’re a developer, I’m not sure what I’ll tell you. But if you’re a DBA, I would tell you exactly what I’ll tell you. Write a blog on backups.\nKN: Only about that subject?\nGF: The reason I say backups is because one of the problems that we run into is that people aren’t taking their backups. It was recently reported that the New York Police Department has a database that they haven’t backed up ever. It’s a database of money, of confiscated materials. And they don’t back it up. You look around and you’re like: “Really? I thought we solved that problem.”\nDW: Yes, like 20 years ago!\nGF: Yeah, I kind of really thought that we had this one cornered. Everybody knows to run their backups. But evidently not!\nKN: Nobody remembers who did it.\nGF: So the thing is like: “Please go out there and write an article on backups.” Because somebody who’s not doing their backups is going to read your article. They’re gonna find your article. And your article is going to speak to them personally in a way that they’re going out there and get their backups done. That is the same way on every single topic. It doesn’t matter that somebody else has done it. Your point of view is going to be different. And your point of view may speak to other people better than my point of view. There’s no reason why you can’t do the exact same article that I just did. In fact, go and steal my last technical article and write your own version. Why not? It’s not gonna be a bad thing to have two people talking about the same topic.\nKN: Basically, in a different way.\nDW: This is crucial what you have said: “Write it in your own way.” It’s not purely stealing.\nGF: Yeah, I don’t want you to literally steal it. That would be bad. If so, I’ll come down and find you.\nDW: That’s right. It always has to be something personal in this article – the author’s point of view.\nDW: What do you feel is your greatest achievement? In your professional or personal life?\nGF: I was gonna say I raised two kids and they’re in college. I think that is my number one achievement. I managed not to break them.\nKN: That’s true.\nGF: I think that’s number one. Other than that – professionally, I think it was when I made an MVP. That was a moment that I will remember forever. It was just amazing. Most recently, I would say – being honored enough to become PASS President. I’m still coming to terms with that one. I might think differently later. But really the one that I made an MVP. I was just so chuffed, amazed and out of my depth. I can’t imagine another feeling like that. It was cool.\nDW: How long are you an MVP?\nGF: I’m coming up on 10 years.\nDW: So it’s like me – I started being an MVP in January 2009.\nGF: So you are ahead of me. Mine was in April. 1st April – April Fool’s Day. I had to check the email header to be sure that it was really from Microsoft. I thought somebody might have been fooling with me.\nDW: Do you have any animals at home?\nGF: Yes, actually I do. We have a little German Shepard puppy. She’s about 9 months old. Her name is Anika. Right now she’s maybe 40 pounds, so she’s very small. She’s probably going to be about 70 pounds or so. She’ll be a good sized dog.\nKN: That’s why I thought it was small because I probably saw it in the picture.\nGF: Yeah, we had another dog – Maximus – and he died. He was a big boy. He was 80 pounds. He was a big buff bear of a dog.\nKN: Thank you very much Grant. At the end of our conversation I would like to ask you where can we find you, where can the listeners can find you?\nGF: Online – @GFritchey is my Twitter handle and my blog is www.scarydba.com. You can find me there. And live – the only thing that I have coming up is that I will be at Live360 in two weeks in Orlando, Florida. But more importantly in February, I will be on SQL Cruise in Caribbean, which is www.techoutbound.com if you’re gonna to check that out. It’s run by Tim Ford. It’s a pretty amazing event. It’s special. You get more dedicated time with speakers there than you will anywhere every. Kevin Kline is going to be there. It’s going to be a great event.\nKN: Yeah, it’s a special place as well.\nGF: Yeah, and also you’re get to hang out on the beach with rum. So that’s cool.\nKN: Yeah, very cool.\nDW: And hopefully we will be able to host you again in Wrocław during SQLDay one more time.\nGF: I would come back to Wroclaw in a second. That was a great event. You guys are doing a great job. Believe me! I’ve sent people to you, like: “Go and ask him if you can speak!”\nKN: Thank you for that.\nGF: I’ve talked up Poland like you wouldn’t believe. I loved it there.\nKN: Damian, now you know why we have so many speakers from the USA.\nDW: That’s great. We’d like to be even bigger because we have a potential to be bigger but it’s community speakers so everybody makes this conference. When you was there last year with us, it was a big kick off for us. We were able to ask Brent and Greg Low to come. They were this year in May. We’ll see who will be with us next year. For example, we have Itzik Ben-Gan who is willing to come and give us all-day workshop.\nKN: But you are always welcome as well.\nGF: Thank you. I would love to come. I’ll have to arrange it with work but I’ll come there in just a second. It’s one of my favorite places now. I didn’t know it was my favorite place till I went there. But now it is one of my favorite places. You and Belgium – I just can’t get enough.\nDW: I’ll send you an email because people were really asking for you. We can have you at least one more time because your sessions were great.\nGF: I would go every year if Redgate lets me. But I know that you will get sick of me at one point.\nDW: It’s knowledge, so people like it.\nKN: Thank you very much.\nGF: No problem. It was great talking to you. Thank you very much. Thanks for the time. And keep doing what you’re doing.\nUseful links: Grant\u0026rsquo;s Twitter: @GFritchey\nGrant\u0026rsquo;s blog: www.scarydba.com\nGrant\u0026rsquo;s books on Amazon\nRedgate: Twitter | Website\nPASS Organisation\n","date":"2018-06-26T23:25:38Z","image":"/uploads/2018/06/IMG_7990__ASF011_PASS_Summit_with_Grant_Fritchey.jpg","permalink":"/2018/06/asf-011-grant-fritchey-interview/","title":"ASF 011: Grant Fritchey interview"},{"content":"Woohoo, a lot of time has passed since this meeting in Chorzow, Poland.\nYes, you have heard right - Chorzow. This second SQL Saturday in Poland was originally registered by my friends for Katowice but finally happened in Chorzow, which is city nearby to Katowice. For me, as always, it was very good occasion to meet whole polish team, but also, learn something new. As I haven\u0026rsquo;t played speaker\u0026rsquo;s role that time - I could have focused on relaxing and listening selected sessions. I had a chance to attend to Stephanie\u0026rsquo;s workshop and learn things about R language. Great workshop for the beginners, thanks Steph again!\nOn Friday, the day before main conference day, 4 full-day workshops have taken place:\nSteph Locke - Introduction to R [EN] Mikael Wedham - A Master class on workload tuning [EN] Tomasz Libera - SQL Server Integration Services w praktyce [PL] Hubert Kobierzewski - Kompletna hurtownia danych w chmurze – krok po kroku [PL] All SQL Saturdays in Poland are organized by DataCommunity Poland.That event happened on 16 September 2017.\nLink to this event: http://www.sqlsaturday.com/664/\nOK, no more words, let\u0026rsquo;s have a look at some photos.\n[srizonfbalbum id=9]\n","date":"2018-06-20T08:00:38Z","image":"/uploads/2018/06/SQLSaturday-Katowice-2017-Cover.jpg","permalink":"/2018/06/sql-saturday-664-katowice/","title":"SQL Saturday #664 Katowice"},{"content":"Auditing in Azure SQL Data Warehouse\nIt\u0026rsquo;s not a common case we\u0026rsquo;re sharing an internal link to this blog. But this is unusual moment: the first guest post written by my friend, Janusz Rokicki [T].\nQuick Recovery Time with SQL Data Warehouse using User-Defined Restore Points\nNow, with this feature, you can make a snapshot before and after important processes on your data warehouse.\nAzure Backup for SQL Server on Azure now in public preview\nPay as You Go (PAYG) service for backups? Great idea - you don\u0026rsquo;t have to have backup infrastructure and having 15-minute RPO meets the backup organisation\u0026rsquo;s SLA.\nUse Azure Data Lake Analytics to query AVRO data from IoT Hub\nThe video presents how to read and manage (with U-SQL) blob content in AVRO format which is produced by IoT Hub.\nClustered Columnstore Index: Massively Parallel Trickle Insert\nEvery time is good to hear from Sunil Agarwal and his team. This time it\u0026rsquo;s all about eliminating last page contention in delta rowgroup by using memory-optimized table and move the data to the target table with suggested chunks of 102400 rows.\nPlansplaining, part 6. Aggregates with OVER\nIt\u0026rsquo;s sixth part of the series about execution plan. In this episode Hugo Kornelis explains how exactly OVER clause works and how to read its execution plan.\nIn the video Patrick (Guy in a Cube) presents how to read data from a flat file and build data model. Check why does it worth edit before load.\nAsk SQL Family - episode 11 is coming\nNext episode will show in few days. Watching this video - it\u0026rsquo;s quite easy to guess who is our next guest. Can you?\n— SQLPlayer (@SQLPlayer) 17 June 2018 ","date":"2018-06-17T22:19:15Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/06/last-week-reading-2018-06-17/","title":"Last week reading (2018-06-17)"},{"content":"The first article in a series on “Security Intelligence in Azure PaaS” inspired me to write something on auditing in Azure SQL Data Warehouse. To put it simply for anyone new to GRC (which stands for Governance, Risk Management, Compliance), auditing is just like logging, but in a more expensive suit and tie. Business tends to understand its purpose and requirements and is aware of the attached price tag. Azure SQL Data Warehouse is relatively new MPP (Massively Parallel Processing architecture) database offering. It is based on Parallel Data Warehouse with Generation 2 (optimised computing) generally available, but some regions and performance levels still need to submit support requests. On the surface, it’s a kind of SQL Server, but behind the scenes it’s a different beast altogether. This might explain the inconsistent experience people have with Azure Portal compared to the more mature Azure SQL Database.\nHow to enable audit? Auditing is disabled by default and the UI experience depends on the region to which the logical server is deployed. For instance, in UK South, the portal offers no options to manage auditing:\nIn North Europe, the portal allows Table Auditing (table-storage based) to be enabled on the SQL Data Warehouse scope, but it isn’t possible to enable Blob Auditing:\nOn top of that, Blob Auditing behaves differently when enabled on a logical server level in different regions. In locations that support Table Auditing, turning on Blob Auditing automatically enables it in all databases, including SQL Data Warehouses—and that\u0026rsquo;s expected. In other regions, Blob Auditing is not automatically enabled and has to be turned on programmatically by calling ARM REST API. The PowerShell script to enable Blob Auditing can be found here:\n1 2 3 4 5 6 7 8 9 10 11 \u0026lt;# https://docs.microsoft.com/en-us/powershell/module/azurerm.sql/set-azurermsqldatabaseauditing #\u0026gt; $setAzureRmSqlDatabaseAuditingSplat = @{ ResourceGroupName = \u0026#34;ResourceGroupName\u0026#34; ServerName = \u0026#34;ServerName\u0026#34; DatabaseName = \u0026#34;DatabaseName\u0026#34; StorageAccountName = \u0026#34;StorageAccountName\u0026#34; State = \u0026#34;Enabled\u0026#34; } Set-AzureRmSqlDatabaseAuditing @setAzureRmSqlDatabaseAuditingSplat Overall Blob Auditing seems to be the best option—it\u0026rsquo;s a common denominator that is available across all regions and SQL PaaS resources, and it is based on the SQL Audit feature with access to specifications that can be adjusted to individual business requirements. Details including audit actions and groups can be found on the reference page (follow the link above). The fact that Table Auditing has already been deprecated in SQL Databases might also help with this decision.\nHow to modify audit specifications? Back in February, I wrote a post describing how to extract the time of the last successful geo-backup in Azure SQL Data Warehouse. The key takeaway was to “trust, but verify”, which I usually recommend in context of security in Azure. With a little help from the auditing process, it\u0026rsquo;s possible to verify quite a lot of managed operations, like backups and DBCC checks—at least in theory. By default, audit specification consists of three groups:\nSUCCESSFUL_DATABASE_AUTHENTICATION_GROUP FAILED_DATABASE_AUTHENTICATION_GROUP BATCH_COMPLETED_GROUP The query to get actions for audited groups can be found here:\n1 2 3 4 5 6 7 SELECT * FROM sys.dm_audit_actions WHERE containing_group_name IN ( \u0026#39;SUCCESSFUL_DATABASE_AUTHENTICATION_GROUP\u0026#39;, \u0026#39;FAILED_DATABASE_AUTHENTICATION_GROUP\u0026#39;, \u0026#39;BATCH_COMPLETED_GROUP\u0026#39; ) And now to put them into practice!\nWhen I queried DMV above for available groups, I noticed DBCC_GROUP. In Azure SQL PaaS, Microsoft is the party responsible for consistency checks and fixing data corruptions, not customers. To live by “trust, but verify”, it would be useful to audit these checks, but attempts to add DBCC_GROUP didn\u0026rsquo;t go well:\nNext I decided to add AUDIT_CHANGE_GROUP (listed as one of the enumerator names), because it\u0026rsquo;s good practice to monitor any changes to auditing and other security controls. They are one of first targets I would expect to see hit when an attacker starts moving laterally, but I got:\nWARNING: The action group ‘AUDIT_CHANGE_GROUP’ is not supported. It will be removed in a future release.\nSo DMVs, messages from cmdlets and documentation is a little hit and miss at the moment. After some research, I decided to add BACKUP_RESTORE_GROUP, DATABASE_OPERATION_GROUP and USER_CHANGE_PASSWORD_GROUP:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 $AuditActionGroup = @(\u0026#34;SUCCESSFUL_DATABASE_AUTHENTICATION_GROUP\u0026#34;, \u0026#34;FAILED_DATABASE_AUTHENTICATION_GROUP\u0026#34;, \u0026#34;BATCH_COMPLETED_GROUP\u0026#34;, \u0026#34;BACKUP_RESTORE_GROUP\u0026#34;, \u0026#34;DATABASE_OPERATION_GROUP\u0026#34;, \u0026#34;USER_CHANGE_PASSWORD_GROUP\u0026#34;) $setAzureRmSqlDatabaseAuditingSplat = @{ ResourceGroupName = \u0026#34;ResourceGroupName\u0026#34; ServerName = \u0026#34;ServerName\u0026#34; DatabaseName = \u0026#34;DatabaseName\u0026#34; StorageAccountName = \u0026#34;StorageAccountName\u0026#34; State = \u0026#34;Enabled\u0026#34; AuditActionGroup = $AuditActionGroup } Set-AzureRmSqlDatabaseAuditing @setAzureRmSqlDatabaseAuditingSplat Next, it\u0026rsquo;s time to verify that the audit actually works and read some events.\nHow to access auditing logs? Blob Auditing uses Append Blobs, which impose a few restrictions on storage accounts (no premium storage, no VNET endpoints), but the advantages seem to outweigh the limitations. For instance, events are produced in near real-time and can be consumed directly in Stream Analytics jobs as input sink.\nOne of the easiest ways to access auditing logs is by using sys.fn_get_audit_file. This TVF is not supported in SQL Data Warehouse and has to be called from the context of the master or any other SQL database under the logical server that hosts SQL Data Warehouse.\n1 2 SELECT * FROM sys.fn_get_audit_file(\u0026#39;https://StorageAccountName.blob.core.windows.net/sqldbauditlogs/ServerName/SqlDataWarehouseName/\u0026#39;, default, default) Recent versions of Management Studio come with an option to extract and filter the auditing log directly from storage accounts (File -\u0026gt; Open -\u0026gt; Merge Audit Files):\nFinally - Blob Auditing uses the .xel format, and log files can be copied across and opened locally in SSMS.\n","date":"2018-06-13T06:53:13Z","image":"/uploads/2018/06/Auditing.jpg","permalink":"/2018/06/auditing-in-azure-sql-data-warehouse/","title":"Auditing in Azure SQL Data Warehouse"},{"content":"Good day SQL folks!\nLast week was pretty hot not because of the weather. Microsoft has bought GitHub and this was the news number one for most of the time. Meanwhile I was attending the ITCamp in Cluj giving two lectures about USQL and RLS Security implementation in SQL Server. Especially the second session went not as usual as there was no electricity in the venue and I was the only one who did the session. I had a portable beamer and two power banks and full room of people.\nNow let\u0026rsquo;s start the Monday with something new to read.\nFree E-Book: A Developer’s Guide to Building AI Applications\nThis book, which is clearly developer-focused, walks you through the process of building intelligent cloud-based bots, and makes relevant code samples available from GitHub.\nDenormalizing your data with Azure Functions and Cosmos DB’s change feed\nIn this article you find how to do a denormalization, a common data processing pattern and how it can be easily implemented using Azure Functions to process the change feed of Cosmos DB containers.\nQuickstart: Run the SQL Server 2017 container image with Docker\nHere you have an introduction of how to start using Docker to pull and run the SQL Server 2017 container image, mssql-server-linux. Then connect with sqlcmd to create your first database and run queries.\n[Video - 1:01:26] We\u0026rsquo;ve built on relational databases for decades, and even used SQL databases in the cloud. But the relational model, as flexible as it is, cannot model every modern scenario we see in our applications. With Cosmos DB, we finally have many kinds of NoSQL database at our disposal, all under one umbrella. Key-value, columnar, document, and graphs, we have many more options to build on.\nPower BI Desktop June Feature Summary\nThis month our reporting features focus on accessibility and more flexibility when formatting your charts. With filtering and sorting options, our data view is now more powerful. Several connectors also get major updates this month, including a new driver for our SAP Business Warehouse connector that comes with numerous significant improvements\nASF 010: Dejan Sarka interview\n[Podcast - 25:30] If you missed that - there is a new interview. This time with Dejan Sarka who is one of the most recognized person in our DB world. At least one of the most colorful.\nCheers,\nDamian\n","date":"2018-06-11T07:30:14Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/06/last-week-reading-2018-06-11/","title":"Last week reading (2018-06-11)"},{"content":"Modern data warehouse\nSee how a modern data warehouse works from beginning to end.\nLoad confidently with SQL Data Warehouse PolyBase Rejected Row Location\nGreat news for all Azure SQL DW folks! Now, loading data into Azure SQL Data Warehouse using PolyBase is more flexible. The process can redirect invalidated rows to a selected folder.\nUsing Email Attachments As A Data Source In Power BI\nHow to load data directly from email to Power BI? Chris Webb [T] reveals it for you.\nCumulative Update #9 for SQL Server 2016 SP1\nThe 9th cumulative update is now available for download at the Microsoft Downloads site.\nHow to securely access PaaS resources with \u0026lsquo;Allow access to Azure services\u0026rsquo; option disabled?\nJanusz [T] shows what is worth to change in default settings after a provisioning within Azure to stick best practices in terms of security.\nDateAdd vs ParallelPeriod vs SamePeriodLastYear; DAX Time Intelligence Question\nIn this post Reza Red [T] explains the difference between SamePeriodLastYear and ParallelPeriod.\nGroupBy June recordings are online now\nGroupBy [T] just finished up another day of free online training. Have you missed it? You can watch the recordings now on YouTube. Enjoy!\n","date":"2018-06-03T22:07:39Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/06/last-week-reading-2018-06-03/","title":"Last week reading (2018-06-03)"},{"content":"With this episode I would like to commence long-awaited new series\nIt\u0026rsquo;s about DevOps with databases. I\u0026rsquo;ll start with SSDT, but not exclude other tools (like Redgate ReadyRoll) in the future.\nStarting from the very basic things, we will travel through entire process to see and understand how to work, develop, tests and maintenance database project within company.\nIn this very first episode I\u0026rsquo;ll introduce SSDT and show up how to create empty database project and how to import existing databases from physical server with 3 methods:\n1) Import from database (connect to server)\n2) Import from DACPAC\n3) Import from script\nLet\u0026rsquo;s get the ball rolling What is SSDT? SSDT - SQL Server Data Tools.\nIt is a free add-in? from Microsoft to Visual Studio which allows you create a database project (like a Desktop Application or Console project in VB/C# but for database).\nWhy do we need SSDT? Have you ever written a code in VB(.NET), C#/++ or other to build application? Imagine that you need to write entire application only in your mind or using notepad. Skipping the fact, that a code was being written in that way (C) and writing HTML code is a different kettle of fish and doesn\u0026rsquo;t count. Hence, using SSDT you have a tool to maintain a database project and it allows you:\nverify query statements (T-SQL) validate data/column types validate compatibility with specific version of SQL Server (SQL Server 2005! - Azure SQL Database v12) prepare repeatable build process compare and find the differences (i.e. between project and database) apply standards \u0026amp; best practices onto your enterprise or customer and eventually, create an automated process of deployment Certainly, there are much more than those mentioned. If you know another one that I\u0026rsquo;ve missed - leave it in comments below, please.\nHow to get it? You have two options:\n1) Choose it from Visual Studio installer. Now it\u0026rsquo;s one of the modules you can simply select during installation.\nHave Visual Studio already installed? No worries. Just run the installer again, do add required module.\n2) Go to the website and download appropriate extension depending from your Visual Studio. Afterwards install it and run Visual Studio.\nHow to check whether I have SSDT on board?\nRun Visual Studio and go to the menu: \u0026ldquo;Help\u0026rdquo; -\u0026gt; \u0026ldquo;About Microsoft Visual Studio\u0026rdquo; Opened window will reveal what modules you\u0026rsquo;ve installed.\nDo find \u0026ldquo;SQL Server Data Tools\u0026rdquo; on the list and you\u0026rsquo;ll find out the version as well.\nOther way is to create new project trying to find \u0026ldquo;SQL Server Database Project\u0026rdquo; node. Can you see it? Good. You can use SSDT within VS.\nNew (empty) database project Once you open Visual Studio - go to the menu: File-\u0026gt; New -\u0026gt; Project.\nNew window will be opened and then find the project type: SQL Server Database Project.\nHaving an empty database project you can either create new objects and create new database from the scratch or import existing one.\nYou can import the databases following one of 3 methods.\nImport database from Data-Tier Application (dacpac file) One of less popular option, in my opinion. You can create dacpac file using SSMS, for example.\nDACPAC file is a zipped file which contains all information about database model. I\u0026rsquo;ll describe it more in separate post.\nAnyways, it\u0026rsquo;s not a perfect method as export DACPAC file might be very tough, especially when your database has got errors, i.e. unresolved references.\nBut let\u0026rsquo;s assume that you already have one.\nRight-click on database node in your new project and select Import -\u0026gt; Data-tier application.\nFind needed *.dacpac file, decide whether you want to import permissions and/or database settings and click Start.\nNew window will be opened and the process commences. The summary of import will be created in a log file.\nImport database from server This kind of import is useful when you have access to the SQL server with database to import. The limitation is that you can do it only once.\nIf you already have some object in the project - this option is disabled. To re-enabled it, you must delete all objects (files) from the project.\nAnyways, I use this option whenever I can. Why? You do not miss any objects or properties and you do not rely on other people who might have done the dacpac file (or script) wrongly. Thus, if I can do something faster and without additional risk, why not?\nRight-click on database node in your new project and select Import -\u0026gt; Database\u0026hellip;.\nIn the Import Database window you need to choose a connection to source database. Even if you don\u0026rsquo;t have any - click Select Connection\u0026hellip;\nPick up one of the existing connections you\u0026rsquo;ve used previously (tab History) or create new one (tab Browse).\nPut a server name (including its instance name when required), select appropriate authentication method and database name (list should be populated when all above information are correct and your access is valid).\nThen do click Connect and you\u0026rsquo;ll be ready to click Start to commence the import process.\nImport database from script (sql file) When you have no access to source server - this is option is for you. The script can be generated by someone else and you can use it to import a model. This option is very handy also in a case when part of a databases has been imported already or if you want to import only selected objects. In other words - this option is reusable.\nFurther more, one or more files might be imported during one pass. All files are being interpreted and the database objects defined in those scripts are added to your project.\nWhat happens once your objects you\u0026rsquo;re importing already exist? Don\u0026rsquo;t worry. SSDT will not overwrite them entirely with a new version. Objects will be merged. Cool, isn\u0026rsquo;t it?\nHow create SQL script of database? Simple. Use our favorite tool - SSMS.\nOpen SQL Server Management Studio, do right-click on the database name, Task -\u0026gt; Generate scripts.\nThen you\u0026rsquo;ll be able to select specific database objects you want to script. Or simply select Script entire database and all database objects option. In the next step (\u0026ldquo;Set Scripting Options\u0026rdquo;) choose between:\nSave script to file and a) create one (big) file or single file per object b) Target directory\nc) Encoding of file(s) 2. Save (copy) script to Clipboard 3. Save to new query window - Open new window with generated script in SSMS\nAlso, you have more control over what the script containing. There is small, shy button Advanced, but very underestimated and powerful.\nBefore you click Next and generate the script, I would suggest to check all options behind the scenes.\nThe following options are very important in our case, so set them as TRUE:\nScript Owner Script Indexes Script Triggers These options also can be important - depends on your database:\nScript Collation Script Change Tracking Script Data Compression Options Why did I say that the Advanced Scripting Options are underestimated?\nDue to \u0026ldquo;Types of data to script\u0026rdquo;. The property, by default, has value \u0026ldquo;Schema only\u0026rdquo;, but you might select one out of:\nData only Schema and data Schema only Data. Yes, you can see it correctly. SSMS CAN script DATA for you. Maybe not in a very efficient way, as it\u0026rsquo;s scripted as many INSERT statements (one per row), but it\u0026rsquo;s free. Bear in mind that the script with data is completely useless in our case. Why? SSDT cannot keep data in a project. How to workaround this, I will explain in a future post in this series.\nWhich an environment should I do import? The best source of the database would be a production server. In most cases, enterprises which didn\u0026rsquo;t adopt DevOps culture - the databases between environments are slightly different. In many places, for many reasons. It\u0026rsquo;s a common scenario, don\u0026rsquo;t worry. But production is production; the environment which the customers, stakeholders work on. If that works (because it works, right?) - this should be your image, your only version of the truth. Afterwards, you can create branches and import rest of environments to reflect their changes.\nImport complete When the whole process comes to the end you will see the summary of import and copy of that information will be created in a log file. All the objects from the source (depend on which way did you choose) have been created in the project under the folders (see below).\nAny questions? That\u0026rsquo;s all in this episode. Do you have any question about above or another topic around DevOps and/or databases? Put them in comments or send via a contact form. And please do remember that next episode of the series will appear soon.\nThanks for reading.\nUpdated @ 08/11/2019\nThis post is already part of the whole series about SSDT.\n","date":"2018-06-01T08:00:44Z","image":"/uploads/2018/05/sunrise-1756274_1280.jpg","permalink":"/2018/06/new-db-project-with-ssdt-and-import-existing-database/","title":"New DB project with SSDT and import existing database"},{"content":" Introduction Dejan Sarka, MCT and SQL Server MVP, is an independent trainer and consultant that focuses on the development of database \u0026amp; business intelligence applications. Besides projects, he spends about half of the time on training and mentoring. He is the founder of the Slovenian SQL Server and .NET Users Group. Dejan Sarka is the main author or co-author of sixteen books about databases and SQL Server. Dejan Sarka also developed many courses and seminars for Microsoft, SolidQ and Pluralsight.\nThis talk has taken place during SQL Saturday in Ljubljana, Slovenia on 9th December 2017 (Saturday).\nWhat is Dejan\u0026rsquo;s favourite sports and which one he cannot plays?\nHow many MVPs were in the world when Dejan was given the title?\nCheck out his last book and find out what he called as a bullshit? Without what he cannot start each day?\nCheck these answers out in this episode.\nTranscript Kamil Nowinski: Hi, Dejan!\nDejan Sarka: Hi, Kamil!\nKN: Thank you for taking my invitation for that interview. Could you introduce yourself?\nDS: I’m Dejan Sarka. I mean like professionally I’m quite an old guy because I am in this business for more than 30 years. For last 22 years I deal with SQL Server from version 4.21 – if I remember correctly. That was the first version I started to work with.\nKN: And at that moment you were doing the Sybase, yes?\nDS: It was already SQL Server but it was the develop in parallel with Sybase. With 6.0 in ’96 I started to specialize for SQL Server only. And I think with 20 years ago we started to give courses about SQL Server. From 2000 I am MVP for SQL Server.\nKN: From 2000? So basically 17 years. Where do you live?\nDS: I live in Ljubljana in Slovenia.\nKN: Here, in this place. How long?\nDS: I was born here. I’ve spent most of my life here except that I lived in the mountains for 22 years in Alp area in the north-western part of Slovenia.\nKN: What are you doing for your living?\nDS: I deal with SQL Server so it means I do consulting, training, writing books and also, I am involved in projects but only if I am interested in them, like some data mining, advanced stuff, statistics and so on.\nKN: So basically, you are an independent consultant.\nDS: Yes.\nKN: I know that you are an author of many books. How many do you have on your account right now?\nDS: I’m finishing the 16th book.\nKN: Are you currently working on the new one?\nDS: This is the one I will finish in a couple of weeks and I am close to have a contract for the next one.\nKN: Which one was the last one? What is the title and the subject of the book?\nDS: The last one was SSIS 2017 Cookbook, one before that was the SQL Server 2016 Developer\u0026rsquo;s Guide – this is the one that we are currently updating to SQL 2017. I am planning the next one about data science with SQL Server.\nKN: So will it be about the R language, Python…?\nDS: R language, Python – all things that you can do advanced analytics.\nKN: What kind of specific tools are you using during your work?\nDS: I prefer for R code I prefer R studio, but for the other development I use visual studio including Python and for T-SQL I still prefer Management studio.\nKN: Have you seen the new one? SQL Operational Studio.\nDS: No, I haven’t. I haven’t played with that one yet.\nKN: Do you like that idea to release the application for multiplatform application?\nDS: It definitely gives you broader market, so why not? I mostly deal with the development side – BI development and this sort of things – so I don’t really care what is the platform behind it. So if I can get new market, why not?\nKN: What do you think about MVP title those days? I know that you have that title for a very long time.\nDS: It’s a group of advanced people, typically very nice because this is more like an appreciation for your community work. So without this community work, without sharing knowledge, a person probably cannot be an MVP. Although I was an old guy, I miss those times when there were just about 17 of us. Everybody knew each other. We were like one a little bit bigger family.\nKN: At that time this award basically started?\nDS: There were some which were already MVP for three or four years. Kalen Delaney, Neil Pike – they became MVPs even five years before me and they were the first two. Even Neil Pike was the first one. Because I remember when you were talking about good old days he was complaining that this program started to be overcrowded when Kalen joined.\nKN: This interview is not only about technical stuff but also about the personal things. Do you have any hobbies? What is your hobby?\nDS: I really do sports a lot. My favourite sport is basketball but unfortunately, I have completely broken knee. I still join my friends when they play basketball – I go to fitness. Then we go for a beer together. But most of my time now I spend on a bike. I also do some running, fitness and hiking. From the “love” perspective basketball is still the first one and since we are talking about basketball, it’s worth mentioning that the Slovenian Team is European Champion in basketball. And from “time” perspective, I spend most of the time on a bike.\nKN: Did you achieve some competition?\nDS: It is not for competition although I am pretty decent biker especially on the slopes but I am not doing this for competition.\nKN: It is just for fun.\nDS: Yes, you know, we are a group that frequently goes together on some longer trips and during warm season we meet at some specific points, pubs, bikers pubs – we meet regularly.\nKN: How would you work your work-life balance? This is a very common topic right now especially for the guys who work as independent consultants. It could be very hard to achieve.\nDS: I would say that in general in Slovenia we are pretty good at it which simply means that we try to work efficiently but also, we try to have a lot of free time. So, if you look at other Slovenian MVPs here: Mladen, Matija, Tomaz, nobody looks too tired, right? I think we are quite successful in managing the balance. Personally, I even try to change the balance in favor of free time – less work, more free time. But if you look at Mladen for example, he doesn’t look worn off.\nKN: How can you do that? Is it because of the experience you have…?\nDS: There are a couple of things. First, it depends on the person. If you are afraid to reject any job, any work, you think maybe the next time they will not call you and you will need them. But then you realize that around the corner there is always a new customer. So rejecting, why not? If I’m already busy, if I already have a full week for like 40 hours, why wouldn’t I reject and why would I work for 70 hours?\nKN: What is currently your limit of the amount of time you spend on work weekly?\nDS: Not so much weekly. You know, I still don’t like to work like 14 hours per day but if I have a course that already means 40 hours and then probably I need to do some writing, so it maybe 50 hours a week. But then I also try to have a few days when I do only writing, like two-three hours per day and then two-three hours of biking, and then five-six hours of beer [laugh].\nKN: What kind of hints would you give to young people who want to start working on the market, like IT world or SQL Server?\nDS: Well, this is a very broad and heavy question – what kind of hints and what is your message to young people. First of all, you know, I’m from a little bit older generation which means whatever I will say now it will not be probably so aligned with current general perception. So one thing is that I never planned my career which is now unacceptable – you should know where you want to be, where you see yourself in two or five years. Who knows? In my opinion it is a bullshit. The most important thing is that whatever you do, you do it professionally, you do it 100%. And then success will come. But also the important part is that you do what you like and what you are talented in. Another big bullshit is that the success is 99% of work and 1% of talent. It is probably the opposite. I am not talented for ballet so 99% of work wouldn’t help me. Sorry. And I would never enjoy it - something that I do that is against my body and mind. You know, my mind is like mathematical, logical. If something is illogical, I cannot accept it – it is really hard to accept it. So I would suffer going in that direction. So why would you go in that direction? Find what you like. What you like is usually the same thing what you are really capable of doing. And then just do it correctly, professionally. Never do it half way – this is most important. Then you see where you get.\nKN: So do you have any daily habits?\nDS: Yes. I really like to start the day with a cup of green tea. This is something that I do even if I have a flight at 3 a.m. – I will wake up at 1 a.m. and have my cup of tea. Otherwise I cannot start the day. I am not so passionate about coffee. I do like it but the morning cup of tea is definitely something that I have to do no matter of what. What other habits? I like to devote time for sport whenever I can. So let’s say, if I have a meeting 5 km away I’ll try to walk there if it’s possible – it I’m not in a hurry.\nKN: How often are you travelling across the Europe or the wide****world giving the sessions for the events?\nDS: On average this is more than one week per month and the maximum is about 10 days per month.\nKN: How are you preparing yourself for a speech?\nDS: First I want to get the idea what I want to say, what I want to explain. I try to make a story out of it. Then I think how I will enforce this story with demos, so then I write the code and prepare the demos. And from this point it is simple. Then I just add slides, maybe some graphics if it has some good idea. That’s it.\nKN: And you prepare just highlighting some special preparation, like repeating the subject, refreshing the idea in your mind before you start session?\nDS: I’m doing this like for 20 years, so it is kind of a part of my life. Whenever I am solving some real-life problem as I’m consulting or whenever I’m writing some books or articles, I immediately see that this is something that it is also worth talking about. So I kind of get the idea for the presentations in advance. And from that point it is usually easy.\nKN: Yes, you have a lot of practice and you don’t need special preparations.\nDS: As I said, typically these presentations come from other work. So I already know the subject, I have already worked on the subject. It typically starts from the other side. When I do something – writing or consulting – I immediately get the idea that something could be interesting for the presentation as well.\nKN: Let’s go back to the books. As I mentioned you are the author of many books. On the market we can download from the Internet a bunch of the free books nowadays. Do you think it is still worth to release the paper books or writing the books?\nDS: There are many positive things here about writing the books. This free stuff is written usually very quickly but not so deeply. There is limited knowledge inside them. I’m not saying that in generally everything is bad but a lot of the time this is just about what’s new in SQL Server, what’s new in that product – the jump start to something. But if you really want to get deeper knowledge about it, it is worth investing in more advanced books. From writing perspective, yes – all of that free stuff is affecting your income. But there are many other positive things. First of all, you learn a lot. You know, whenever you do any development, any consulting, you always focus very narrowly on the problem and once you solve it, you think that’s it. You think you solved every problem in the world. When you write the book you really need to check everything to get the full story to, to check all possibilities not just…\nKN: Yes, it broadens your perspective.\nDS: Yes, so this is how you get to really broad and also learn the deep knowledge which is very, very rare. Typically, you have people with broad knowledge or people with deep and narrow knowledge. To get broad and deep knowledge, this is what writing a book is about. And this is a big advantage in the current world. You know, there are so many over-specialized consultants that write about solving things with a single tool. Or there are those who are more like already sales people. They know a little bit about everything. So it gives you at the same time broad and deep perspective. And what’s more, the paper book is still the best marketing tool – there is nothing better. A presentation or a conference like this, it’s great – maybe you are a good presenter, maybe you have a good day or have a perfect presentation, but the next one after you is the best presenter ever, so people will simply forget you – and all of your effort is lost from marketing perceptive. Books will be there on shelves for 10 years. You will more before they will replace it and throw it away. Recycle… So generations will see it and generations will know your name.\nKN: And that way you make that they will remember about you.\nDS: Exactly.\nKN: Are you still learning something new about technology or people yourself during such conferences?\nDS: Definitely. I still try to learn. Of course, not that much as I used to. I’m in the state that I forget more than I learn, but I still try to. You know, first I want to see what the new technologies are, where the world is going to and then I pick up what looks like something that I could like. And then I try to learn it.\nKN: What do you think about this extension with the analytics language, like R and Python in SQL Server engine?\nDS: I think it’s quite a good idea. They solved a couple of big problems with these data mining, data science machine learning projects. If you look especially at R, Open Source, typically it is not scalable, not very secure. So by having this in SQL Server we these scalable library and SQL Server Security – it’s finally enterprise ready. Same thing about Python. It is not so specifically analytical language, it is more general language. But for analytical part it gives really a lot of power to existing Python developers or even more to SQL Server developers for which R is maybe overkill. Having possibility to write some code in Python and get this still very huge power of powerful algorithms and with all of the security and scalability it is really useful for many people. So I think that finally this is enterprise ready. I’m not saying that data mining in analysis services is not enterprise ready but it is limited. It is limited with the number of algorithms. Is has all specific language – DMX (Data Mining Extensions) language – so it has limited audience. You have millions of R developers, millions of Python developers and millions of SQL developers. Now you can combine all of these worlds. And solving this problem with security scalability. So it seems like all of this data science is enterprise ready but also ready for even small and medium businesses. If you can have all this in express edition, why not?\nKN: This is a good thing, right?\nDS: Yes, it’s a great thing!\nKN: So now all of those data scientists can migrate to SQL Server and use the new versions, use the Python or R.\nDS: All SQL developers can switch to or improve their skills.\nKN: Yes, with the courses from Microsoft to achieve the data science certificate.\nAt the end of our conversation – because we are sitting in one of the rooms at SQL Saturday in Slovenia, Ljubljana - tell me when did this conference start?\nDS: The first SQL Saturday in Ljubljana was 5 years ago. This is the 5th one.\nKN: From my perspective, I think that this is the one of the most popular SQL Saturdays in Europe, yes? Maybe even in the world, because some people from the USA are coming here.\nDS: Yes. We have international speakers and even the audience. I guess, it’s quite popular.\nKN: Do you remember how many submissions do you have this year?\nDS: More than 150.\nKN: All the best then. Thank you for having me this year.\nDS: You’re welcome.\nKN: Thank you very much. And thank you for accepting my invitation.\nDS: Thank you.\nKN: Thanks, Dejan.\nUseful links: Dejan\u0026rsquo;s Twitter: @DejanSarka\nDejan\u0026rsquo;s latest books:\nSQL Server 2017 Integration Services Cookbook\nSQL Server 2016 Developer\u0026rsquo;s Guide\nOther books\n","date":"2018-05-31T06:00:17Z","image":"/uploads/2018/05/IMG_5313-ASF010-SQLSaturday-with-Dejan-Sarka.jpg","permalink":"/2018/05/asf-010-dejan-sarka-interview/","title":"ASF 010: Dejan Sarka interview"},{"content":"Good day SQL folks!\nMonday morning is always tough to start working on serious tasks. I think it is a perfect time to read something interesting before you really go into tons of tasks. Now let me give you some things to read for the rest of the day.\nMicrosoft SQL Server channel on YouTube\nAnnounced by Bob Ward and I think I will be a frequent guest there. You should also rememeber that there is also a great channel about the Power BI as well.\nSet up Azure Cosmos DB global distribution using the SQL API\nIn this article you will see how to use the Azure portal to setup Azure Cosmos DB global distribution and then connect using the SQL API. There are two main things described - how to configure global distribution using the Azure portal and how to configure global distribution using the SQL APIs.\nModernize and extend your ETL/ELT workflows with SSIS activities in ADF pipelines\nAs we continue our march towards General Availability (GA) of SQL Server Integration Services (SSIS) in Azure Data Factory (ADF), we are excited to announce the release of Execute SSIS Package activity as the most direct and code-free method to invoke SSIS package executions within ADF pipelines.\nHOW A SSIS LOOKUP WORKS DIFFERENT FROM A T-SQL JOIN (SSIS SERIES)\nWith a Lookup in a dataflow in a SSIS Package you can achieve a similar result as with a T-SQL Join clause in a SELECT statement: you can check if a row exists in a different dataset, based on one or more (business / natural / surrogate) key columns. However there are a number of differences.\nAnnouncing SQL Advanced Threat Protection (ATP) and SQL Vulnerability Assessment general availability\nSQL Vulnerability Assessment (VA) provides you a one-stop-shop to discover, track and remediate potential database vulnerabilities. It helps give you visibility into your security state, and includes actionable steps to investigate, manage and resolve security issues, and enhance your database fortifications. VA is available for Azure SQL Database customers as well as for on-premises SQL Server customers via SSMS.\nCheers,\nDamian\n","date":"2018-05-28T12:30:20Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/05/last-week-reading-2018-05-28/","title":"Last week reading (2018-05-28)"},{"content":"That was a really tough week for us. Damian and I have led a workshop at SQLDay conference in Wroclaw (Poland) and also a regular session (separately this time). I will publish all materials soon and Damian has take a lot of good pictures, so let\u0026rsquo;s fingers crossed to see them on the blog soon either. That\u0026rsquo;s why not too many reading during past week.\nCreating Azure SQL Managed Instance using ARM templates\nA recipe how to cook it with ARM template and PowerShell script.\nExtract management insights from SQL Data Warehouse with SQL Operations Studio\nInterested to see more information about your Azure SQL Data Warehouse (SQL DW)? New widget shows you Data Skew, Columnstore health and statistics, Users Activity and Resource Bottlenecks. This is only beginning.\nAccelerate your cloud data warehouse with automation tools\nSupposedly, there are 3rd tools that let you automate some work with SQL DW.\nLONDON PUG DATA VIZ - Episode 2\nPrathy [T] has recently started a new discipline to encourage attendees of London Power BI User Group to make your own dashboard based on the same dataset.\nQuick Tips For Debugging Large Stored Procedures\nRead all tips from Brent Ozar\u0026rsquo;s Team which help you cope with loooong Stored Proc.\nBookmarks and Buttons: Making Power BI Charts Even More Interactive\nIn that post, Reza [T] shows you some usages of buttons and bookmarks which is related to visual’s interactions.\nIncremental refresh in Power BI Premium\nIncremental refresh enables large datasets in the Power BI Premium service with several benefits. In the video, Adam Saxton [T] and Christian Wade [T] discuss the significance of incremental refresh.\nA brief reminder of the SQLDay conference (stay tuned - there will be more\u0026hellip;)\nThat's true - speaking at #SQLDay makes people happy ;) @DamianWidera @NowinskiK #DataCommunityPL #SQLFamily pic.twitter.com/WBxkRkuf55\n— Agnieszka Cieplak (@AgCieplak) 14 May 2018\r","date":"2018-05-20T21:22:38Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/05/last-week-reading-2018-05-20/","title":"Last week reading (2018-05-20)"},{"content":"Good day SQL folks!\nSQLDay has just started! We have a workshop about DevOps and Kamil promises to publish a great series of posts about SSDT and topics around that. We have an idea that the DevOps topic should be alive as this is the growing trend in the IT.\nNow let me give you some things to read for the rest of the day.\nAzure SQL Data Warehouse now supports automatic creation of statistics\nAzure SQL Data Warehouse (Azure SQL DW) now supports automatic creation of column level statistics. Azure SQL DW is a fast, flexible, and secure analytics platform for the enterprise.\nAnnouncing new Async Java SDK for Azure #CosmosDB\nA new asynchronous Java SDK for Cosmos DB’s SQL API is now open sourced on GitHub. This SDK leverages the popular RxJava library to add a new async API surface area for composing event-based programs with observable sequences. It also features an improved user experience and is also lighter weight than our previous synchronous Java SDK (yielding a 2x performance improvement on the client-side)!\nEvent Hubs for Kafka ecosystems\nEvent Hubs for Kafka ecosystems provides a Kafka endpoint that can be used by your existing Kafka based applications as an alternative to running your own Kafka cluster. Event Hubs for Kafka ecosystem supports Apache Kafka 1.0 and newer client versions, and works with your existing Kafka applications, including MirrorMaker.\nCosmos DB Solves Common Data Challenges in App Development\nWhen considering how to implement your application with a relational database, it can change how you build it dramatically. Some of these challenges include adding an abstract implementation of the schema in the code, mapping data to objects, building queries, and preventing SQL injection attacks.\nOptimization Thresholds – Grouping and Aggregating Data, Part 2\nThis article is the second in a series about optimization thresholds related to grouping and aggregating data. Check also Part 1.\nAnd by the way: SSMS 17.7 is now available. Go, grab and use it!\nCheers,\nDamian\n","date":"2018-05-14T10:53:54Z","image":"/uploads/2018/05/32381113_10214362446068080_4687022579477643264_n.jpg","permalink":"/2018/05/last-week-reading-2018-05-14/","title":"Last week reading (2018-05-14)"},{"content":"Good morning all,\nThis week only a few things as the SQLDay conference will commence next week and we both (Damian and I) are preparing our workshop and regular session as well. As little time left as so many things left to do!\nBlazing fast data warehousing with Azure SQL Data Warehouse\nGeneral availability of Azure SQL Data Warehouse #Gen2 now supports up to 128 concurrent queries with intelligent caching.\nQuick round-up from GuyInaCube [T], a few words about few posts.\nAzure SQL Data Warehouse and Azure Databricks: Now Even Better Together\nHow Azure SQL DW can cooperate with Azure Databricks and how to build modern big data analytics platform? A quick review of few scenarios.\nSecure credential management for ETL workloads using Azure Key Vault and Data Factory\nWith Azure Key Vault, you can encrypt keys and small secrets like passwords that use keys. Azure Data Factory is now integrated with Azure Key Vault.\nBig Data and Analytics: Incorporate Intelligence into your ApplicationsFollow these interactive scenarios to find out how incorporating AI into your applications can improve your bigdata and analytics.\nConcurrency Week: How to Delete Just Some Rows from a Really Big Table\nDeleting a batch of rows might be painful. Check this Brent Ozar\u0026rsquo;s post how to relieve this pain.\n","date":"2018-05-07T08:51:11Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/05/last-week-reading-2018-05-06/","title":"Last week reading (2018-05-06)"},{"content":"Good day SQL folks!\nWe have an entire week off in Poland. Well, not quite off because you would need to go to work Monday, Wednesday and on Friday but most teams have already taken days off. This is a great spring break after intensive four months of work. I hope that maybe some of you still have time to read this weekly post and be up to date what we think is important in the Data Platform world.\nAzure Cosmos DB query cheat sheets\nNot a blog post or article but it is useful. The Azure Cosmos DB query cheat sheets help you quickly write queries for your data by displaying common database queries, operations, functions, and operators in easy-to-print PDF reference sheets. The cheat sheets include reference information for the SQL API, MongoDB API, Table API, and Gremlin/Graph API\nHow to connect and perform a SQL Server database restore from Azure BLOB storage\nHaving things in the cloud should make life simpler but I have experienced it’s not that straightforward. Once all access / configuration is sorted out then yes, of course! But in the meantime, it can be tedious (even frustrating) and the end result is something that could have been achieved with a different method.\nSQL Server 2017: How to Get a Parallel Plan\nShort, but an informative article written by Dmitry Piliugin about when and how SQL Server generates a parallel plan. There is an entire collection of Dmitri\u0026rsquo;s articles and they are really worth of time reading.\nReplicated Tables now generally available in Azure SQL Data Warehouse\nAnnounced earlier last week that Replicated Tables, a new type of table distribution, are now generally available in Azure SQL Data Warehouse (SQL DW). SQL DW is a fully managed, flexible, and secure cloud data warehouse tuned for running complex queries fast and across petabytes of data\nConcurrency Week: How to Delete Just Some Rows from a Really Big Table\nA blog post from Brent Ozar about how to effectively delete data from a table where the TRUNCATE command is not an option.\nBonus for you at the end - GitHub repository with client library that allows Azure SQL DB or SQL Server to act as an input source or output sink for Spark jobs.\nDon\u0026rsquo;t forget to listen to our interview with Pedro Lopes!\nCheers,\nDamian\n","date":"2018-04-29T08:41:26Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/04/last-week-reading-2018-04-29/","title":"Last week reading (2018-04-29)"},{"content":"Redgate Shift LEFT magazine\nThe latest issue of Redgate’s Database DevOps magazine explores the benefits of DevOps for CEOs, CIOs, and IT Managers, and takes a comprehensive look at the new General Data Protection Regulation, providing a valuable insight into how organizations can meet its demands.\nAzure Cosmos DB query cheat sheets\nI\u0026rsquo;m preparing a separate site on this blog only for cheat sheets, but in the meantime try this out.\nDAX Best Practices\nMarco Russo\u0026rsquo;s recorded session from SQLBits 2018 conference.\nTesting database connections with ssisUnit\nMy friend Bartosz [T] from family country has written a post about ssisUnit tests for connections (this time).\nDear SQL DBA Q\u0026amp;A: A Poorly Indexed ISV Database\nKendra [T] advises what should you do if you’ve got a poorly indexed database from a commercial software vendor?\nDynamically Changing A Chart Axis In Power BI Using Bookmarks And Buttons\nChris [T] demos how to change what is displayed on a chart axis dynamically using Bookmarks and Buttons without remodelling and writing DAX code.\nPower BI Custom Visuals Class (module 99 – Drill-Down Donut Chart)\nThe short video presents advantages and how to use that custom visualization.\nhttps://www.youtube.com/watch?v=yJnmyVFfKCY\n","date":"2018-04-22T23:06:44Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/04/last-week-reading-2018-04-22/","title":"Last week reading (2018-04-22)"},{"content":"The ability to query and make use of historical data is very important for the end users. We are obliged to keep the historical data for some period of time. It depends of the kind of a business or a project. Sometimes it could be months but it could be several years as well. The shortest history I have ever kept in the data warehouse project was two weeks. The database was just too big as I loaded over 300 millions of rows on a daily basis. And it was many years ago.\nSo far we have had to prepare such queries but it requires an additional programming work. One might use triggers, stored procedures or other features like Change Tracking or Change Data Capture to achieve the goal.\nThe triggers and stored procedures are present in all editions of the SQL Server. The seems to be easy to create and use. But you might get into trouble if the base table changes its structure. That means that the maintenance of them might not be that easy as it seems to be. In general changing the base table structure leads the developer to change the trigger or the stored procedure body and of course to alter the table that contains the historical data. This operation could be risky and of course, the altered object needs to be tested.\nWhat has changed in the SQL Server 2016? There is a solution and it\u0026rsquo;s called temporal tables. That is exactly what we need. The definition taken from the MSDN page is as follow:\n\u0026ldquo;A system-versioned temporal table is a type of user table designed to keep a full history of data changes and allow easy point in time analysis. This type of temporal table is referred to as a system-versioned temporal table because the period of validity for each row is managed by the system (i.e. database engine).\u0026rdquo;\nLet\u0026rsquo;s try to do some demos. In this article, I am going to prepare a sample database and two temporal tables.\nHere is the database definition - I know, it is just a regular database\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 USE master GO IF EXISTS(Select * FROM sys.databases where name=N\u0026#39;TemporalDemo\u0026#39;) BEGIN alter database TemporalDemo set single_user with rollback immediate drop database TemporalDemo END GO CREATE DATABASE TemporalDemo GO In the second step I will prepare two tables. The first table will be constructed that way that the corresponding historical data table is created by the SQL Server.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 CREATE TABLE dbo.Products ( ProductId INT IDENTITY NOT NULL PRIMARY KEY, Name NVARCHAR(100) NOT NULL, Number INT NOT NULL, Price MONEY NOT NULL, SYSSTART DATETIME2(1) GENERATED ALWAYS AS ROW START NOT NULL, SYSEND DATETIME2(1) GENERATED ALWAYS AS ROW END NOT NULL, PERIOD FOR SYSTEM_TIME (SYSSTART, SYSEND) ) WITH ( SYSTEM_VERSIONING = ON); GO The table contains columns to store the regular data but you can easily identify two new columns:\nSysstart will be used to determine the start time of the period Sysend will be used to determine the end time of the period The period is defined in the PERIOD FOR_SYSTEM_TIME phrase Both of the columns must be not nullable and be datetime2 type of any precision.\nThe magic started as I declared the table with SYSTEM_VERSIONING=ON. That creates a temporal table which can be understand as a pair of tables: the base table that holds the current state of the records and the historical table that stores the changed data. For every data change we will see a corresponding entry in the historical table. Tthe historical table name was chosen by the SQL Server and it looks like this:\n1 MSSQL_TemporalHistory_For_ObjerctId The historical table has been created in the same schema like the base table. What is also interesting is that the historical table has PAGE compression turned on which seems to be very good choice in most cases.\nLet\u0026rsquo;s now take a look at the historical table structure. It is exactly the same as it is in the base table. However the clustered index is different:\n1 2 3 4 5 6 7 8 9 10 11 12 13 CREATE CLUSTERED INDEX [ix_MSSQL_TemporalHistoryFor_565577053] ON [dbo].[MSSQL_TemporalHistoryFor_565577053] ( [ProductId] ASC, [SYSSTART] ASC, [SYSEND] ASC ) ON [PRIMARY] GO I will dig into that topic in the future posts but it is now worth to mention that:\nThe base table must have a primary key defined The historical table will not have the primary key The historical table will have the clustered index defined on all columns that are part of the key on the base table plus the two system columns described earlier Clustered columnstore index can exist in the base table Partitioning can be applied on the base table but the schema will not be propagated on the historical table. You have to do it manually – which seems to be a good option The historical table cannot have any constraints: primary key, foreign keys and constraints are forbidden Truncate table will not work on historical table if the SYSTEM_VERSIONING = ON DML operations are forbidden on the columns that describe the period of time The historical table must be placed in the same database Let’s now check if we could control the process of historical table creation. It is of course possible, and we can control the historical table schema and name as well as we can required additional logical checks that could be done during this operation. I will describe that process in separate blog post, too. Here is the second table I will be using in my demos\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 CREATE TABLE dbo.ProductsV2 ( ProductId INT IDENTITY NOT NULL PRIMARY KEY, Name NVARCHAR(100) NOT NULL, Number INT NOT NULL, Price MONEY NOT NULL, SYSSTART DATETIME2(1) GENERATED ALWAYS AS ROW START NOT NULL, SYSEND DATETIME2(1) GENERATED ALWAYS AS ROW END NOT NULL, PERIOD FOR SYSTEM_TIME (SYSSTART, SYSEND) ) WITH ( SYSTEM_VERSIONING = ON ( HISTORY_TABLE = dbo.ProductsV2Hist ) ); GO After we created the second table we now see that the name of the table is controlled by us but the structure of the table is exactly the same like it was in the previous example. The only change which maybe is not seen at the first glance is that the historical table is not compressed by default.\nThis is all for today. I will create some more tables in the second part of this series. You will see the different scenario in that case. Now we start from a very simple point, where you start with an empty database and just add temporal tables to the database\nStay tuned\nCheers,\nDamian\n","date":"2018-04-22T21:25:29Z","image":"/uploads/2018/04/P1833376.jpg","permalink":"/2018/04/starting-from-zero-temporal-tables-part-1/","title":"Starting from zero - Temporal tables, part 1"},{"content":" Introduction Pedro Lopes is a Senior Program Manager in the Database Systems Group, based in Redmond, WA, USA. He over 15 years of industry experience. He is currently responsible for Program Management of database engine features for in-market versions of SQL Server, with special focus on the Relational Engine.\nHe also works closely with several Tier 1 SQL Server customers, as well as partners in the field, to understand and gather information about product usage and feedback to drive improvements into the product. Prior to this role, he was a Premier Field Engineer based in Europe for several years.\nHe has deep understanding and experience in various features of the product spanning several versions of SQL Server, starting with SQL Server 7 and up to and including SQL Server 2016.\nThis talk has taken place during PASS Summit in Seattle, WA, on 3rd November 2017 (Friday).\nWhy working for the Tiger Team is so fascinating? Why Pedro started to love the data? How to start to be a community leader? Is sharing knowledge an important thing? What does Pedro proud of the most?\nCheck these answers out in this episode.\nTranscript Kamil Nowinski: Hello Pedro.\nPedro Lopes: Good morning Kamil, how do you do?\nKN: Very good, very good.\nPL: Nice to see you again.\nKN: You too, thanks! So this is the last day of PASS Summit. We are in the Microsoft Clinic stand. Could you introduce yourself?\nPL: Sure, so my name is Pedro Lopes and I am a Program Manager with the SQL Server Tiger Team. For those that don’t know the Tiger Team owns the in-market versions of SQL* and we can act with the customers to make them successful and platform which means doing a lot of reactive work – fixing bugs, issuing service packs, but also doing the proactive work helping shape architectures, write best practice articles and we also own a lot of documentation for in-market version. So in nutshell, that’s what I do as part of that team.\nMe specifically, I own the relational engine for the in-market versions which means everything that makes your queries “tick” if you will. And I mainly enjoy performance troubleshooting and performance tuning.\nKN: Ok, tell me where are you living? You are working for Microsoft in Redmond, yes?\nPL: Yes.\nKN: Where are you living? In some area?\nPL: It’s close. It is 12 miles from here. It was snowing this morning actually, which is nice…\nKN: Yes, it is freezing this morning.\nPL: It was actually snowing there which is not common where I live. But it was a nice surprise\nKN: OK.\nPL: Yes, but I am living close enough to Redmond campus.\nKN: OK, tell me how long or from which version do you work with SQL Server?\nPL: Oh gosh\u0026hellip; So I have been working with SQL Server only since 2002. Actually, it’s been some years now. Coming to think about it. But when I started, I started with SQL Server 2000 but I had still SQL Server 7 laying around. And I guess that were a couple of SQL Server 6.5 laying under some desks, so basically that’s how I started.\nDamian Widera: Pedro, how do you think how tough or how easy it is for you to adopt to new versions of SQL Server, because right now we have a new SQL Server let’s say every two years, and maybe even more frequently?\nPL: Yes, so definitely the release cycles are shortening. If you think of it, it only took around 16 months between the time we released SQL Server 2016 to releasing SQL Server 2017.\nDW: Exactly.\nPL: And that has to do like it was said in the keynote and in several other sessions, that has to do with keeping up the pace of innovation, right?\nIf you would have to wait three years, four years for a new version that comes with everything that has been developed in the previous years, you as a SQL Server user, are losing a lot of what is happening in the data world. For example, using R algorithms over the data on SQL Server. You do not have to move data out. That is extremely powerful.\nOr now to use the interesting use of polybase. So imagine if it take years to be able to adopt to what is becoming an industrial standard, and you as a company using SQL Server would be left behind. So indeed, times are changing in that perspective\nKN: Yes. And the IT require very fast adopt the new features\u0026hellip;\nPL: True. We grant you that there are let’s say a few frictions. For example, when we changed, when we implemented the new cardinality estimator in the SQL Server 2014 and people were upgrading, that generated some frictions. So for the majority of workloads that actually work very well, but there are few occasionally regressions and we understand that. Now with SQL Server 2016 and now 2017, and the use of the Query Store, and the automatic plan regression that’s taken care automatically for you. So you as a production DBA don’t need to really care that much about that specific anymore.\nSo you can upgrade more confidently, let’s say from SQL Server 2012 or 2014 to SQL Server 2017, take advantage of Query Store using automatic plan regression, and if any regressions happen you do not need to decide “oh, should I go to the trace flags or setting that turns my server back to old CE”. No. Whatever query has issues they will be fixed by using a previously known good version of the plan. But that leaves still something on the table for you as a DBA, because you still need to go in, understand (using a number of tool as I have demonstrated during my sessions here), find out what’s not good and why it’s regressed. And then do the necessary work with your developers to make sure the query works natively in whatever is the new compatibility level.\nBut the point is that in production you don’t need to live with the nightmare that maybe a part of my important workload regressed, and now what do I do?\nKN: Yes. So from that perspective we will need the developers but we will not need the DBAs?\nPL: I disagree. Because here is the thing. It is, in my perspective part of the DBA… So there are two main aspects here:\nYes, if you have developed your career doing a performance tuning some of the more menial tasks might be the taken care of for you. Now for example, the practical result of regressed workload, let’s say when you move from SQL Server 2012 to SQL Server 2017 is, let’s say 20% of my workload. Because sometimes two wrongs make right, it worked better, although it was not necessarily a better plan, but it worked better from a standpoint, in a previous version. And now moving to the SQL Server 2017, because that can be detected and corrected automatically, you as a DBA don’t need to do that part. But you still need to provide a root cause for the change. You still as an experimented DBA, need to provide mitigation steps and actually more than that. Resolutions – to make sure that the queries you take, tune them to work natively, in let’s say SQL Server 2017 version or whatever you are using, and then you will be able to do away with the mitigation that automatic plan regression implemented because now you are running – in the better shape – in the native version of the compiler.\nKN: So the DBAs should not be worried about their future?\nPL: I don’t think they should. And by the way. With all those new technologies coming in to the SQL Server ecosystem like polybase and R and others, you have more diversity. So, it is an ever changing world – yes, but it’s also a world that allows you to learn something new and keep yourself relevant, if you will.\nKN: Yes.\nDW: Yes, and this is a great explanation because for example, what I have heard from customers and people that occasionally just look at the system, they were thinking that really we don’t need the administrators. But it is exactly like you explained. The role is changing, is evaluating, and it might not be simple tasks in a moment. They will be different tasks. So maybe you don’t have to know everything about tuning, because you have now better tools, but other things still, or other features still require DBA and people who understand what’s going on in the system.\nPL: Absolutely. I agree with you.\nKN: Pedro, tell us, what specific tools are you using during your work?\nPL: To be honest I am a huge fan of SSMS. Being a user of the SSMS for years obviously, and I have used tools from ours partners, which are doing an excellent work in leveraging all the diagnostics artifacts we are putting into the engine with their own tools is great, we are not competing with anyone – let me be very clear. But with all the, because I work a lot with performance tuning and query performance troubleshooting, I am big user of the SHOWPLAN and many things that evolves around making SHOWPLAN a one stop shop for everything you need to know about the query execution. The context on which it executed, because we have added information about statistics loaded, we have added information about the trace flags active to the system, everything you need to know about the context of the execution of a query. But then we have added a lot of performance metrics of each operator in the plan. So all of that is… we’ve been working with customers, with users, and we ourselves are users of the tools in order to troubleshoot and help our own customers. So right now we’re releasing a new tool as you know.\nKN: Yes, I wanted to ask about it. It will not replace the SSMS, yes?\nPL: Right away no it will not. Let’s be very clear. So, our investments will go into that multiplatform tool, absolutely. We are not at a point, because the tool (which was actually explained yesterday by Ken who is the engineering manager in the team that owns that tool), that the source code will be published to GitHub. And we will call upon the community to help improve the tool because it’s basically a front end for a collection of microservices. So at this point of time, for example for the scenarios I was talking about, which is deep query troubleshooting and some tools that we have embedded over the last years to the SSMS, they are not available in the new tool. They will be in the due time and that can happen faster or not, also with the community helping. Because users with enough skills to know what they want out of the tool, will be able to contribute to its success. I think it makes a much better outcome. So, in a nutshell, the SSMS will not be deprecated yet, because there is no \u0026hellip;\nKN: Replacement…\nPL: … complete replacement. But yes, it’s true that our investments will be further onto the new tool, because it’s multiplatform, right? And because it can leverage help from the community, whereas SSMS…\nKN: Remind me the name of the new tool?\nPL: It’s Microsoft SQL Operations Studio. Sorry, I am still getting used to the name.\nKN: Yes.\nPL: It is MSOS. **\nKN: Yes, MSOS.\nPL: I know the acronym. It is the Microsoft SQL Operations Studio.\nKN: So for this application everyone can be engaged in developing this project?\nPL: Yes, so the source code will be made available in GitHub shortly. The preview as far as I know start in mid-November. We will release a preview so everyone can start to use it. And again, it’s the rich environment when the user will point out what he needs out of the tool, maybe it’s comparable with SSMS and maybe it’s getting something new that SSMS never had. And being able to provide a feedback and better yet if he has a skillset to actually submit changes to GitHub and improve the tool on his own, then let the community leverage all of that. I think that’s the best world we can be in.\nKN: Yes, it’s really great idea I think.\nDW: OK, Pedro, what do you think about MVP those days. About the program and people. How the see it?\nPL: They are all great partners. That’s what I can tell you. We have, as you might now, a direct feedback channel from MVPs, because we share a DL (distribution list) with them. The MVP community is both a hardcore user of SQL Server that pushes SQL Server to its limits. They in fact, many MVPs are themselves consultants as they work for our end customers in terms of SQL Server. They do provide a bridge to those scenarios and to the feedback that’s stands from that. And then, just giving us honest feedback, that we use in order to improve the product, because that is how we have been doing it for more than two years now. We build what the users want to use. We don’t build anything hoping someone will adopt it.\nKN: Exactly, that will not make any sense, that approach.\nPL: Yes.\nKN: Let’s talk about you as a person. Not only from a professional view but also from a personal point of view. What is your hobby? Do you have any hobby?\nPL: Hobbies. Yes, I do. I am huge movie fan. I devour movies. I have my own collection of movies. Blu-rays, DVDs… I am an old school. I have never gone to the fad of downloading movies from the internet. I really go and buy movies on blu-rays and DVDs.\nKN: It’s still not very old school. If you told me you have tapes aor something like this that would be very old school.\nPL: People say it’s strange that I do that and not just download the versions that’s made available or just sign up to Netflix or Hulu or something like that. No, I still like to get, to sort them alphabetically, yes, it’s like an index sitting in my shelve. Over 1600 DVDs and Blu-rays. Yes, that’s one of the thing I do. And I also collect a movie memorabilia.\nKN: And what is the movie memorabilia?\nPL: Artifacts that were other used in films or are representative of films like statues, or props used in movies, or stuff like that.\nKN: Oh, ok.\nDW: So that’s really a hobby and I think also your passion…\nPL: Absolutely.\nKN: Yes, it’s also work like this. What do you think about your work-life balance?\nPL: My what?\nKN: Work-life balance. This is a currently very common phrase. You know – how to …\nPL: Kamil, thank you, let me drop you there. I know what a work-life balance is. I was just… It’s variable. It varies. Sometimes it balances more to the work and sometimes it balances more to the life. That’s what I can tell you.\nKN: Ok, fine.\nDW: So I expect you are a very busy man also, working for Microsoft that must be a lot of tasks every day you have to deal with\nPL: So like I said, I work in a team that is outwards facing. Our main mission is to help customers become successful with SQL Server. That takes a lot of facets if you will. That includes for example, a customer open cases with the CSS and they find bugs, for example in some edge cases. Let’s remember that SQL Server runs on millions of servers, with billions of different workloads, and it needs to adequate, adapt to all of them. Now it’s bound that you find edge cases where the server misbehaves or even hits bugs, right? So that’s the part of the work we need to work with those customers to make sure that we unblock them as fast as possible, and make them successful with the platform they are using.\nBut also in, for example, in learning from those interactions and seeing what we can implement at tools level or the engine level, to make sure that once you hit a blocker for you, which might not be a bug, and may be a performance issue or something like that, you have the tools and the artifacts in the engine to quickly identify and take you away from the problematic place, right? And that’s a huge part of my work, specifically as I said in the beginning, within the Relational Engine.\nDW: So it must be really very fascinating because you have feedback from people that are using it, using the engine, the SQL Server. They use it in a very different ways, and you see how the work done by Microsoft is consumed by people. So probably you will never get bored at work.\nPL: Yes, I can tell you, there’s never a boring day.\nDW: Yes, I can imagine. My question would be that, imagine that you are speaking to students. What would you tell them as you know they are going to start their careers in IT market? What should they choose? Do you have any advices for them? How should they start, how should they build their careers?\nPL: That’s a tough question and I don’t have a definite answer. If I can give any advice, would be stay open minded, be conscious that even if you have 20 years of career or just starting you will learn every day, and be open to learn every day. Technology changes around us at an incredible pace. And keep up, find your space and thrive in it. Leverage the work of others. Collaborate, work as a team. I guess, keeping up the students’ spirit throughout the rest of your career. I think it would be my advice here. What technical area to choose? I personally love databases but I don’t expect everyone to work with that.\nKN: Let me rephrase the question then. How you started as a young man in the IT, and or maybe how it happened that you started working for Microsoft?\nPL: Oh my god…\nKN: I hope you remember that…\nPL: Yes, fairly well. I started working with SQL Server by accident. I always working with technology, I started when I was in high school. Then moved to a college. But the point is that actually in college I majored in something different but then went back to IT and to computer programming. What I was going to say is I, I’ve got to working with databases by accident, starting actually with SQL Server 2000 and early versions of Oracle, and then I found that I loved to being able to model data, to answer business questions, by understanding how to model data to provide those answers, and I got hooked ever since. Working for Microsoft, well I am on my eight year at Microsoft, and as you would, I saw an ad and I applied and ran my course of interviews, and one leads to the other, and eight years later I am here. Working with stuff I love every day.\nKN: Ok, great! You are making a lot of speeches across the conferences. So how you prepare yourself for the speech.\nPL: There are two main schools of preparedness if you will. There are those who prepare a month in advance and they lock down their sessions and demos, and everything is lock down a month beforehand. I don’t belong to that club. However, I do keep an updated, let’s say, master collection of topics I am passionate about and I like to talk. And when I am preparing to a conference I pull them together let’s say two weeks before. Maybe I’ll refresh the demos, and sometimes I refresh the demos minutes the session starts.\nKN: Oh, no\nPL: Because sometimes you have something you would like to show and do those. I rehearse the demos from time to time just to make sure they flow, but sometimes impromptu to I just show some more detail that I was anticipating, or something also depending on how the questions I am getting and how people reacting on them. So, there is a some degree of preparedness and some degree of improvisation.\nKN: Damian, any comments, question?\nDW: Yes, yes, because I also do some speeches as well and nowadays I cannot somehow imagine that you can be prepared like months before for a conference. Because everything is changing especially in the cloud. So in the Azure you could be very surprised that there a lot of new features you have not expected to be there. So I think we have to be very …we have to change things almost every day or a week. In best case every month. But maybe we can ask you about something else. Do you have any animals at home?\nPL: Do I have any what, I’m sorry?\nKN: Animals.\nPL: No, I don’t, my kids are begging me for a pet dog and I am almost breaking, so possibly very soon.\nDW: Dogs you know are very good for kids. I have two and two cats and they are playing together\nPL: Yes, but they don’t take care about themselves.\nDW: No…\nPL: Now that my kids are old enough to be responsible. The tradeoff will be if getting a pet, you are mainly responsible for them. If they accept that then ok.\nKN: They are playing together and destroying together everything\nPL: yes, that I hope won’t be happening that often, but will cross that bridge if it comes to it, Kamil\nKN: Ok, the next question is which achievements are you satisfied the most, of your private or professional life?\nPL: Oh, I am super proud of my kids and everything they do every day. That’s on the personal aspect the greatest achievement. And I think their mother, my wife, has much more credits on that than I do. Having said that professionally I can’t put my finger on one thing. Again, I am working with a fantastic team and fantastic people. I am learning every day and hopefully I am contributing something. I really can’t point to anything specific. Every release, every CU, I’m proud that we are doing the work that customers expect from us. And that’s what fulfills me at the end of the day.\nKN: I can notice that you use the Twitter as a way of communication. Are you fan of Twitter or why you choose that way of communication?\nPL: Well, in that case I am not a… I am just a follower. So I know that community is fairly active on Twitter. Especially during the conferences. And it gets immediate attention but it also fades away very quickly. I would say I am not a heavy user of Twitter. I use it as another means to try to evangelize the work that we have been doing, and with that tried to help people to take an advantage the improvements they may expect from the work you are doing.\nDW: Ok, I would like to get back to some work stuff. What do you think about passion at work? Because from my perspective, if you love what you do at work and it is really a passion you really can have many, many successes, almost every day. Do you think the same?\nPL: I would agree. You have your personal satisfactions if you will. I loved when we were able to show the world Plan Comparison and Plan Analysis. And I loved when we were able to demonstrate to the community, the ways they might use the small changes that we have put in, because the community demanded us to. And that drives a sense of satisfaction.\nKN: Is this an answer to your question Damian?\nDW: Yes, of course!\nWe have a great community in Poland, DataCommunity, which also organizes a SQLDay conference as you can remember. And is there any specific advice you could say to people who would like to start doing their talks during the community meetings? Because we have a lot of leaders like Kamil, me and other great guys in Poland but sometimes we see people that would like to tell something but they somehow don’t want to go on stage and talk to other people. Do you have any advice? How to start? How to not be scared?\nPL: I would say, start by talking that is towards to you. If you work with the technology and you and you faced challenges before, share with the community how you overcome them. Others possibly have faced the same, and they would welcome the lessons that you have learned and how you have overcome them. I would say that’s an excellent way to start sharing with the community and about a stage fright, that it is something I cannot help with to be honest. Just take a deep breath before the plunge and go for it. The community is like a SQL family. No one will grill you for attempting to share with the community. Everyone will welcome that. So don’t be afraid from sharing what’s close to you. What you think others can leverage, because in all honesty, if you faced a challenge before, it’s possible that others out there are now facing that challenge, and you will be able to help them.\nDW: Yes, and that’s our role as leaders. We have to try to build the community and support possibly new leaders or people who like to share the knowledge. My thinking about it that people sometimes don’t want to speak, don’t want to talk to other people because they think their knowledge is not at the appropriate level.\nPL: I would say that this is human – thinking that. I would also challenge you to say “if you don’t share and gauge your knowledge with others, then how can you learn if you don’t expose yourself?”. Not locked in the basement\nDW: Exactly, that’s the point.\nKN: At the end of our chat I would like to ask you where we can find you? Where people can find you?\nPL: Oh, so twitter is a good place to start. Both through the @mssqltiger twitter handle and my own @sqlpto (actually now @SQLPedro). There is a contact form in our blog for example which comes directly to our team, so those are good things to do it.\nKN: Yes, we put those links of the twitter and the blog to the post. Thank you very much for the conversation.\nPL: Thank you Kamil.\nKN: Thank you.\nPL: See you next time!\nKN: Cheers.\nDW: Thank you! Have a good day!\nPL: You too, bye!\nFootnotes * Since we interviewed, we now own all SQL Server, in-market and vNext.\n** Actually it drifted towards SQL Ops Studio. It’s what users referred to. ?\nUseful links: Pedro\u0026rsquo;s Twitter: @SQLPedro\nTwitter of the SQL Server Tiger team: @mssqltiger\nMSSQL Tiger Team Blog: aka.ms/sqlserverteamTiger Team GIT Repository: tigertoolboxMicrosoft SQL Operations Studio: website\n","date":"2018-04-18T22:24:56Z","image":"/uploads/2018/04/IMG_7962__ASF009_PASS_Summit_with_Pedro_Lopes.jpg","permalink":"/2018/04/asf-009-pedro-lopes-interview/","title":"ASF 009: Pedro Lopes interview"},{"content":"Good day SQL folks!\nSpring is everywhere all around us! You might not necessarily see this Monday morning at work\u0026hellip; I hope you will have an easy start of the week with our weekly pack of great articles. Not necessary articles\u0026hellip; Today I have posted more video links and also one GitHub code example.\nIngest, prepare, and transform using Azure Databricks and Data Factory\nNice article about ETL/ELT and what and how you can do this having Azure Databricks generally available with Azure Data Factory.\nCommon design patterns with Azure Cosmos DB\n[Video, 15:47] Aravind Krishna stops by to chat with Scott Hanselman and take a look at common design patterns for building highly scalable solutions with Azure Cosmos DB. They are talking about data modeling and how to choose a proper partitioning key plus some common patterns like time series data.\nHow is SQL Server Statistics Histogram Computation Used by The Query Optimize\n[Video, 10:22] First video out of 3 about insights of the SQL Server. You can check how the histogram computation (SQL Server Statistics Data) used by the Query Optimizer for cardinality estimation. Explanation how this is related to the EQ_ROWS \u0026amp; AVG_RANGE_ROWS information in the statistics.\nChanging Cosmos DB Write Locations on a Schedule with Azure Functions and Managed Service Identity\nAzure Cosmos DB can replicate a single account\u0026rsquo;s data to as many Azure\u0026rsquo;s regions as you desire. Applications around the world can read from the closest location to minimize latency. Write operations, however, are still sent to a single designated write region. However, this can be changed - read how to do this.\nHow to use ETAG to control Replace or Read Document in CosmosDB\n[CODE] This is actually a code sample posted on GitHub. You have an example of how to do basic CRUD operations in CosmosDB\n[Video, 9:55] Brent Ozar talks about \u0026hellip; a few bugs with “actual” execution plans on SQL Server 2017 and SSMS 17.6\nCheers,\nDamian\n","date":"2018-04-16T07:32:32Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/04/last-week-reading-2018-04-15/","title":"Last week reading (2018-04-15)"},{"content":"April 2018 is the 101st episode of T-SQL Tuesday – and it\u0026rsquo;s my second contribution in the series (damn it, I\u0026rsquo;m still working on my constant frequency of writing)\nThis month’s topic comes from Jens Vestergaard (B|T). He is asking us about our essential SQL Server Tools and please be aware that SSMS and SSDT do not count.\nSome time ago I collected most important tools I use in my daily routine at work. You can find the list here (it\u0026rsquo;s time to re-verify it, I guess).\nToday, I\u0026rsquo;m going to pick up only few of them and spread good words.\nSSMS Ah\u0026hellip; yes, I know - I\u0026rsquo;ve just mentioned that SSMS doesn\u0026rsquo;t count, right? Even thought new version is on the market for some time - I still see people who aren\u0026rsquo;t aware about it. So, please stop using SSMS 2014/2016 - simply go on website and download latest version of SSMS.\nSOS No, I\u0026rsquo;m not calling for help\u0026hellip;\nNo further than 6 months ago, Microsoft announced new tool: SQL Operations Studio.\nIt\u0026rsquo;s a child, in preview version and still developing. Perhaps it hasn\u0026rsquo;t so many features as SSMS, but what is really cool (for Mac users the most) that the tool is multi-platform, which means working on Windows, MacOS and Linux. Who would have thought\u0026hellip;\nAnother extraordinary thing is a capability to create your own insight widget writing queries and build visualizations.\nLast but not least - the entire code is open source and you can contribute to development!\nSQL Sentry Plan Explorer This tool is just AWESOME! Especially if you want to find out where a culprit of bad-performance is hiding behind your stored procedures and you aim to tweak a bad query.\nThe stored procedure you\u0026rsquo;re suspect gonna be splitted out onto queries in order to focus on a specific query which causes a trouble.\nThe execution plan brings more information and it\u0026rsquo;s more readable than that in SSMS. Further more, Index Analysis tab provides you with a full picture of a selected operation: columns covered by indexes, statistic histogram and much more. Everything in one place. Perfectly, isn\u0026rsquo;t it?\nAdding opportunity of keeping all history of executed queries to this - you will be loving troubleshooting with it.\nAnd all that jazz is completely FREE from a certain date.\nRedgate SQL Search Those guys have got absolutely fantastic tools (and they are fantastic itself as well).\nSQL Search is totally free tool which helps you to bridle your databases.\nAnd if you are tied with DevOps thingy - you\u0026rsquo;re probably familiar with ReadyRoll, DLM Automation, DLM Dashboard. Working as a DBA (or accidental DBA like me previously) you should know (or at least heard about) SQL Monitor. Perfect tool to keep your eye on SQL Server environments.\nData Script Writer My own simple tool written in C# with DevExpress components to help scripting the data from static or dictionary tables. That kind of script you can easily put into your SSDT project, add post-deployment script and in that way run it many times to deploy data into a database during a deployment. Very handy and useful (I didn\u0026rsquo;t find anything better). There will be separate post about such approach.\nThe BROWSER True. Working with the cloud or web application we use our browser. Microsoft Azure Portal offers dozens of features/systems were among of them are Azure SQL Data Warehouse, Azure SQL DB, Cosmos DB, Data Lake which are in an area of my interest. Obviously, you can (and you should) automate some work with PowerShell, but still, the browser will remain as a \u0026ldquo;first-aid tool\u0026rdquo;, not only for monitoring.\nKeePass 2 Do you still struggle how to keep the passwords safety?\nSo many times I saw that people were keeping these sensitive information in\u0026hellip; plain text using notepad. Are you kidding me? These days? Wake up, GDPR is coming and the chain is as weak as its weakest link. Fix it.\nTotal Commander Do you remember the times with MS DOS? I remember! (yes, I\u0026rsquo;m so old even if I don\u0026rsquo;t look like)\nI used Norton Commander those days. When Windows 95 has come the application changed out into Windows Commander and once it becomes more popular, they had to change the name to Total Commander due to Microsoft\u0026rsquo;s request.\nIt\u0026rsquo;s a fantastic tool to manage the files. You can work with keyboard only, find wanted files pretty quick, zip/unzip, connect to FTP, filter object list, synchronize two locations, compare files by content and much, much more.\nOne of my English friends is laughing that only Polish and Russian guys use this application. Hard to say\u0026hellip; so, if you native English and use the app - let me know.\nHey man, the topic was about SQL SERVER Tools only! Right. My fault. Forgive me. That\u0026rsquo;s the result of sitting and writing a post after the midnight. Not always good idea. Anyway, it\u0026rsquo;s good opportunity to mention about other tools either.\nThanks for reading!\n","date":"2018-04-10T01:48:24Z","image":"/uploads/2017/09/tt_300_11FF5A1D.jpg","permalink":"/2018/04/tsql-tuesday-101-my-essential-sql-server-tools/","title":"T-SQL Tuesday #101: My Essential SQL Server Tools"},{"content":"Good morning folks,\nWe (me \u0026amp; Damian) are preparing our workshop which will take place at SQLDay conference in Wroclaw (time is ticking, sooooo fast), but in the meantime I\u0026rsquo;d like to share with you some articles I have read past week. Enjoy a reading and have a nice week!\nA Brief History of Databases\nMaybe not so brief, but very demonstrative demonstrational. :)\nAzure SQL Data Warehouse costs vs AWS Redshift\nAlthough the comparison between cloud services is not very obvious, Thomas LaRock [T] have tried do that as simple as possible.\nRob Sewell talks about dbatools on RunAs Radio\nListen to the chat which Rob [T] has given. It\u0026rsquo;s about dbatools, Continuous Integration, CD, containers and more\u0026hellip;\nUse read-only replicas to load balance read-only query workloads (preview)\nUse your AlwaysOn replicas of Azure SQL DB (in Premium tier) for read-only application workload.\nAzure Databricks Overview\nCheck Microsoft Azure Databricks out and you will be able to improve and scale your analytics on a global scale.\nCreating a Striped Disk on Your Azure VM\nWould you like to have 4200 IOPS using Azure Storage Containers? Read this trick from Jens Vestergaard [T].\nAzure Cosmos DB Advanced SQL Queries\nAndrew Liu [T] presents advanced techniques working with Cosmos DB. Look how to ask for the same using LINQ Query, LINQ Lambda and SQL.\nPower BI Desktop April Feature Summary\nTake a look at new features in Power BI. One of them is new Q\u0026amp;A experience for consumers (Q\u0026amp;A Explorer).\nhttps://www.youtube.com/watch?v=W_Nb73Od_AI\n","date":"2018-04-09T00:22:29Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/04/last-week-reading-2018-04-08/","title":"Last week reading (2018-04-08)"},{"content":"This part of blog is for a people who never had contact with programming or Power BI. I’m want to show you that is impossible understanding this amazing tools without knowledge about programming. In these place I will teach you step by step how to understanding, start using, and have fun with Power BI.\nSo let’s start First you should know that at these moment we have three kind of Power BI:\nPower BI Desktop it makes us connect to data, shape that date and the last we can use that model to create reports. You can save your work in Power BI Desktop file format. Files will be shared like any other file.\nPower BI Servicewhere you can share your report with another people, publish report and get access your date anywhere.\nPower BI Mobile here you can connect you data and monitoring your business by phone.\nReady\u0026hellip;Steady\u0026hellip;Go\u0026hellip;\nAt the moment we will be using Power BI Desktop. In these post I will show you how to installing Power BI Desktop in your machine.\nIn first step you should installing Power BI application on your machine. Free version MS Power BI Desktop you can find here.\nIn this step, you can register, however, it\u0026rsquo;s not necessary as long as you use only Power BI Desktop not publishing reports to the Portal.\nUsing Power BI Desktop When you download and install Power BI Desktop - first what you will see is Welcome window.\nIn the Welcome screen you can directly Get Data, see Recent Sources and **Open Other Reports.**When you close the window the Report view of Power BI Desktop is displayed.\nAfter opening the BI power we will be presented with a ribbon with three tabs:\nHome- we will find here the most important for us options such as: Get data, enter data, edit queries\nView- we can adjust the size of the report to display in the phone, set the position of the page,\nModelling - allows you to format data, add columns, tables, sort etc\nThe following screen shows the three view icons along the left of Power BI Desktop:\nReport - we can see here the final process of loading data in the form of graphs. Reports may appear on one or several pages.\nData – there is a table with downloaded data, which can be appropriately changed, formatted, to selected charts. These data can come from many sources of Excel, SQL Server, Internet etc\nRelationship – we find here a full data model including links between data tables.\nVisualizations\nOn the right-hand side we can see the visualizations panel where are the types of graphs in Power BI, as well we can formatting and model data from the field bar.\nVisualizations bring your report to life. Thanks to them you can create amazing and completing visual presentations. Below, I describe a list of available out of the box visualizations in Power BI. You can add them to report simply by click.\nVisualizations available in Power BI:\nLine charts, Maps:Basic maps, Maps: Filled maps, Donut charts, Pie charts, Scatter charts, Combo charts, Area charts, Line and stacked column chart, Line and clustered column chart, Waterfall chart, 100% Stacked column chart, 100% Stacked bar chart, Clustered column chart, Clustered bar chart, Funnel charts, Gauge charts, Card charts, Multi – row card, KPIs, Slicer, Tables charts, Matrix charts, Bubble charts. I will show you more details about how to use the most popular charts in the future posts.\nThe next post will be about creating a model and ingestion it in Power BI.\n","date":"2018-04-06T22:40:09Z","image":"/uploads/2018/03/gaston-roulstone-473478-unsplash-1000.jpg","permalink":"/2018/04/how-to-start-with-power-bi-desktop/","title":"How to start with Power BI Desktop"},{"content":"Good day SQL folks! It\u0026rsquo;s April Fool day! But don\u0026rsquo;t worry. All the cool stuff you find here is real \u0026amp; has been read already.\nThe Challenge Is on: Maintain a Custom Letter-Based Sequence\nThe challenge presented by Itzik Ben-Gan who will be also our guest during the SQLDay 2018 conference in May in Wr0cław. As Itzik stated this nice little puzzle was presented by a student Nicolas Barreau.\nReleased: Microsoft OLE DB Driver for SQL Server\nPedro Lopes, who will be our guest in the ASF 009 podcast has informed that Microsoft OLE DB Driver for SQL Server is ready. And you can go and download it. This is a 3rd generation driver of the OLE DB series for the SQL Server. It has multi-subnet failover capabilities.\nTuning performance in Azure SQL Database\nLearn about using recommendations to improve Azure SQL Database query performance\nCardinality Estimation Process\nI love how Dmitry Piliugin writes his articles. He goes to a very deep level of the engine. That requires a lot of knowledge and experimenting as well.\nPower BI Licensing Walk-through Guide\nAll about licensing in Power BI. All you need to know but yet been afraid to ask. Reza Rad is the best person to explain all the details.\nCheers,\nDamian\n","date":"2018-04-02T00:29:55Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/04/last-week-reading-2018-04-01/","title":"Last week reading (2018-04-01)"},{"content":"SQL Server on Linux: overview\nLearn how to get started using #SQLServer on #Linux. Watch the demos via @ch9\nColumnstore Indexes on Standard Tier of Azure SQL DB\nSeveral days ago, Microsoft announced Columnstore indexes for Standard databases in the S3 and above pricing tiers. Niko says \u0026ldquo;check\u0026rdquo;.\nHow Microsoft Does DevOps – an Interview with Aaron Bjork, Microsoft Visual Studio Team Services (VSTS)\nMany valuable advice if you want to adopt DevOps in any large enterprise. Deploy the value as much as every 3 weeks and get feedback faster.\nUnlock your data’s potential with Azure SQL Data Warehouse and Azure Databricks\nHave a glance how to utilize a power of the Databricks Runtime and build advanced ETL processes in the Modern Data Warehouse.\nPower BI Roadmap Announcements In the Dynamics 365 Spring ‘18 Release Notes\nChris Webb and his helpful summary of Microsoft\u0026rsquo;s Power BI team who announced a new capability coming soon to Power BI. Incremental refresh is one of my favourite as it should appear a long time ago.\nIntroducing Azure SQL Database Managed Instance\nLara Rubbelke explains and demonstrates all features of Azure SQL Database Managed Instance, and tell you how you can set it up on this @MSFTMechanics episode.\nAzure Cosmos DB query cheat sheets\nUsing Cosmos DB from time to time? These things gonna help you.\nHave a look how our conference looked like in 2017.\n","date":"2018-03-25T23:50:07Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/03/last-week-reading-2018-03-25/","title":"Last week reading (2018-03-25)"},{"content":"Good day SQL folks! Thanks for visiting our blog again! Here is the information you need to calmly start the new week.\nIndexing and partitioning\nAn interesting example was done by Andrew Pruski when ALTER TABLE SWITCH might actually fail when there are no identical indexes on the partitions. Read this blog post to find the solution.\nControlling Access to Azure Cosmos DB\nThe title says it all. How to control access to your Azure Cosmos DB by Marcin Policht.\nCosmos DB access control with users and permissions\nThis article dwells a bit on the scenarios when to use permissions instead of masterkey and how you can use a set of permissions to grant access to multiple documents, collections, partitions at once.\nDemo – Azure SQL Database Managed Instance in action\nAzure SQL Database Managed Instance is new data service in Azure cloud that exposes fully-managed SQL Server Instance that is hosted in Azure cloud and placed in customer VNET. Here you can see some of the key new features that are available in Managed Instance.\nPower BI Troubleshooting Checklist\nIf you have ever started a sentence with “I’m seeing a strange issue in my Power BI Desktop report” and wondered how to investigate, this article is for you.\nASF 008: Michael Rys interview\nAn interview with Michael Rys we did last PASS Summit. If you haven\u0026rsquo;t heard it yet - now you have your chance\nCheers,\nDamian\n","date":"2018-03-19T02:35:37Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/03/last-week-reading-2018-03-18/","title":"Last week reading (2018-03-18)"},{"content":" Introduction Michael has been doing data processing and query languages since the 1980s. Among other things he has been representing Microsoft on the XQuery and SQL design committees and has taken SQL Server beyond relational with XML, Geospatial and Semantic Search. Currently he is working on Big Data query languages such as SCOPE and U-SQL when he is not enjoying time with his family under water, on the ski slopes, or at autocross.\nThis talk has taken place during PASS Summit in Seattle, WA, on 2nd November 2017 (Thursday).\nDo you ever wonder how many people work on the USQL or Azure Data Lake? What animals does Michael see from his window?\nListen to this podcast to get to know what two career choices Michael had had when he finished PhD in Switzerland.\nTranscript Kamil Nowinski: Hi Michael.\nMichael Rys: Hi.\nKN: Thank you for accepting my invitation for that podcast. Basically this podcast is not only about the technical stuff. It is also about to show you as a person, your habits and other things, what are you doing on a daily basis, etc. So let’s start and could you introduce yourself to the people?\nMR: Sure, so my name is Michael Rys and I am a Program Manager at Microsoft currently working on Azure Data Lake and our internal Big Data Platform called Cosmos. Not to be confused with Cosmos DB. I have been with Microsoft for almost 19 years, started in December 1998 originally working on XML support in SQL Server and then I have worked on SQL Server stuff until about 2011-2012 doing the beyond relational scenario in SQL Server 2012 when we shipped things like FileTable, Semantic Search, more spatial support, xml indexing and couple other things. Then I moved to the Big Data space back then.\nKN: Where are you living exactly?\nMR: I currently live in a small suburb of Seattle and Bellevue called Sammamish which is about 20 minutes’ drive from the Redmond campus…\nKN: I know where it is. My friend lives there.\nMR: Ok, cool.\nKN: Because you mentioned you are from Germany…\nMR: No, I am from Switzerland.\nKN: Oh, Switzerland. I was thinking for a while that you are living and working remotely…\nMR: No…\nKN: So you are a local guy here…\nMR: Yes, I am local, living here for 19 years.\nKN: From which version of SQL Server you are working with?\nMR: So I basically joined the SQL Server team for SQL Server 2000. So I was working on the first set of the XML features that we added in SQL Server 2000.\nKN: Good to know… So you were behind the scenes…\nMR: So you can blame me for FOR XML and OPENXML and some other stuff.\nKN: Good to know and if we had some question not only about the U-SQL so we can ask you about the XML and other stuff.\nMR: Yes.\nDamian Widera: But Michael you are not involved in the JSON stuff which works in SQL Server\nMR: So the JSON stuff in the SQL Server – I did talk to the people when we started adding it to give them a little bit of background information about the design decisions that we made with XML because JSON is very similar to XML in the terms its hierarchical format. I wanted to kind of give them a little bit of a hint to make it similar so people that know how to process one hierarchical format can very easily apply the knowledge to operate on the other hierarchical format as well. But I was not directly involved. That was actually done by our team in Serbia.\nDW: Ok, but I can see the consistency.\nDW: So it is really easy. If you have any experience in playing with XML in SQL Server no matter in which version than it is easy to adopt the previous knowledge to work with JSON. So that’s fine! We don’t have to learn new stuff.\nMR: That’s kind of the idea that was around that\nDW: I would like to ask you – do you have any favorite set of specific tools you are using during your daily work? I mean when you have to play with SQL Server or USQL or Data Lakes?\nMR: You mean besides Outlook?\nDW: Yes, of course.\nMR: Well, as a Program Manager of course some of the things that I do a lot is one things is helping customers with their questions about how to do certain things, how to debug certain things, how to optimize certain things. And also obviously preparing demos for both blog posts, documentation or presentations. And for USQL right now my favorite tool for doing that is basically the IDE extension in Visual Studio. So that’s where I would spend most of my time in like looking in job graphs trying to figure out what people are doing, to help them improve their query performance or improve their query efficiency and of course their cost efficiency. So some of the things I would look for example is the vertex execution view and the modeler are the two things I would often use to help people improve the cost efficiency and understand the jobs. Another thing that I obviously do when I do my own development is basically to use the Visual Studio development experience for doing the USQL and code development. Otherwise, one thing, one tool that I would like to get to know better is our VS Code tooling around Data Lake. So I am kind of planning of probably doing one of my next demo project is to use just VS Code instead of Visual Studio for doing that.\nKN: I noticed that there is an extension for USQL.\nDW: Yes, there is.\nMR: So these are the things that I am currently using and obviously the other things are like a Notepad++ for looking at results and understanding encoding and documents etc. Otherwise it is just the standard program management tools like Skype and Outlook and One Note and Excel and things like that.\nKN: and you are working with USQL and Cosmos and what do you think what is the current direction of all those stuff?\nMR: Well, one thing you notice is that there are a few kind of trends that are occurring not just within our offerings but also in the competing offerings. You will see that people are getting away from the low-level clustering management and low-level programming paradigms and they go into something we call “serverless PaaS offerings” like Azure Data Lake Analytics is and the ability to operate at the higher levels of abstraction. So the serverless PaaS offerings is to manage the costs better. Like you do not have to manage clusters, you don’t have to pay for provisioned things that you are not using. You basically have shared resources that you have if you need them. But if you don’t need them they are basically available for other people to use. It becomes like a self-managed backend system. Another thing that I see is that the higher level of abstraction in terms of programming goes into both going towards SQL-like languages for doing the processing like you saw that with MapReduce when it went to Hive. You see that with Spark when it’s going from the LINQ-like extension for Scala and Python and Java towards like-Spark-SQL. You see it happening also at the level of what people actually would like to do with it the data. It is going away from just doing SQL like analytics into doing Machine Learning, into doing more domain specific processing of the data because the data becomes more unstructured when you operate in these environments. So want to have more domain specific code that is scaled out. So in Spark you see it with the Spark ML having that. In USQL you see it with our extension libraries with the whole framework extension …\nKN: Cognitive Services\u0026hellip;\nMR: Yes, Cognitive Services in this aspect and so on. But also the framework we provide to people who do it themselves.\nDW: Yes, I am thinking about a little bit different thing, because I would like to ask you what role do the MVP play in the – form your perspective – how MVPs can improve for example the technology you are responsible for?\nMR: So, I mean the MVPs have always been very instrumental in couple of ways in the technologies I have been involved in SQL and now in the Azure Data Lake and USQL. I see them as one early adopters. I mean I expect MVPs to often be like people that are using our things at the beginning, give us feedback and basically are the first external customers so to speak like to give us feedback and then use it. Also potentially contribute things to it like especially with the extensibility module that we have – we have some MVPs that provided let’s say an Excel extractor. Tillman did that recently.\nDW: Yes.\nMR: So that kind is one role. Another role is of course around amplifying the messaging. Like go and teach because if it just be us teaching then it would be fairly… as we would not scale as well. So having MVPs helps us with the teaching and bring their own standpoint. And I bring the feedback that I hear from them is another way. And another way is to build community engagement by running user groups not just for our specific products but for the whole data platform aspects as well. And also help customers to understand how these different components that we have in our ecosystem in the Azure fit together and what are the sweet spots to do so. And last but not least obviously it is also basically helping amplifying our messaging for example by inviting me into the podcast.\nDW: That’s right.\nKN: Ok Michael, tell us something about you. Tell us something about your hobby please. Do you have any hobby?\nMR: Yes, I have a few hobbies like you were in the presentation today and you noticed that it was a green car on the race track. So that is one of my hobbies to take my green car into the auto courses and try to improve my driving skills. Another one is scuba diving although I have not got too much time recently in the last years to do that and in the winter I like to go skiing and spend time in the mountains. That’s few of the major hobbies that I have. If I am outside of writing about USQL scripts and so on.\nKN: And very popular topic that is recently – work life balance.\nMR: Yes.\nKN: What do you think about it? How you can manage that?\nMR: Yes, that’s always a hard question. The more fun you have at work the more work life balance you have in term of yourself. But if you have a family than you have to make sure that there is time for family as well. Let’s say this way. At Microsoft at least, there is an understanding that people should have a work life balance so there is a little bit more agreement around that that we might be out of the weekend and not necessarily available online except for if you are on the call rotation. Which is a little bit more work life balance than let’s say if you are in the startup or so. But it still is a, I would say when compare to normal job, if you are a passionate about your work then there is more work…\nKN: at home…\nMR: at home and there is more work as part of your life.\nKN: Yes, so the passion\nMR: so you always have to like a trade off a little bit.\nKN: But it’s also hard to explain this to our wives.\nMR: Yes, but I think that yes, you have to take time off at some point. Either by taking some vacation or reserving let’s say weekend to like spend time with the family and so on. But yes, if you would talk to my wife she would be probably think than I am first married to work… But from the other hand we are still married so…\nKN: So it works. It is good. But you are living very close to Redmond. It’s about 15 minutes by car.\nMR: It’s about… let’s say - it’s about 20-25 minutes.\nKN: Depends on the time\nMR: And depends if there are police officers on the way\nDW: But you have a sporty car so it should not be a problem.\nMR: Yes, the car is not the problem.\nDW: That’s right. Ok, so maybe a question like this. What hints would you give to young people who wanted to start working on the market?\nMR: You mean to start working…\nKN: on the IT market\nDW: as the IT guys…\nMR: Yes, that’s an interesting thing. The first thing I would say is work on understanding your customers. Like regardless there are a lot of interesting areas in IT but I think that the important part is to be successful in the understanding what problems do your customer try to solve. And then understanding what are the best tools that help you solve it. And obviously you wouldn’t have to have some foundation in terms of understanding the technology and the technologies base and potentially different offerings in that space like maybe become an expert in Azure and Azure Data Lake – that’s what I would say. But also Spark definitely would be something if you want to be in the Big Data space. And then find basically the vertical customer segment that you feel you can understand well and be passionate about and kind of provide solution in. That’s if you want to be on the like IT side. The you can bring that into ether as becoming an employee of a company in that space or becoming a contractor or consultant in that space. If you want to go into actual building the system then I think what helps again is understanding the customers etc. but also understanding the technology. And we have MVPs that ending up being a part of the product team. For example not just because they are passionate about the technology but also contributing to the technology and the go through the interview process and basically understand how to be successful there. So it is all, I think the most important part is that you understand in what area are you interested and passionate about and what areas do you think the customers can benefit from your understanding and passion and capabilities.\nDW: so you said a very important thing because apart from the technical knowledge you have to have some communication – how to communicate to customer, how to solve their problems and this is something that can be very tough for young people. They have to learn a lot I think from the older colleagues to have the ability…\nMR: I would not disagree with that, yes, there are obviously ways of how like they can learn that like they can engage in the community events and to speak or just to attend meetups and like to listen to people talking about the problems and so on. Yes, it is more than just building code. I mean when I actually went into the IT area when I was 19 years of age I learned programming at high school. I kind of liked it but then I started using it as a making some summer pocket money. I worked for some local companies and the university as a programmer. And you quickly learn that your programming skills are need definitely to how solve the problems. But understanding the problem you actually need to code for, understanding the customer demands, being able to communicate with the customer to translate his “do what I mean” into actually telling him what he means and being able to actually build the solution is almost as important if you really want to be more than just, what I call, a coder. I mean you can find the niche where you basically code what somebody else tells you to code but I am not sure that something that is fully fulfilling at that point.\nKN: Yes, but this is ok when you are working as a developer and you are very close to the customer like in that case you are describing. But in the biggest company or let’s say – the corporation – nowadays. I think that the developers are doing separate things and BAs (Business Analysts) doing separate things. BA is contacting with the customers.\nMR: Yes, but even we at Microsoft, like me as a Program Manager, so I am not writing the code by myself. I am translating the customer demands into design spec and then the engineering team can implement. But even in the engineering team the engineers are normally providing better solutions if they really understand the customer problem and can communicate back to me and others about what they want to build so we can assess if the solution is actually going to help us address the problem or not.\nKN: Yes\nMR: So I think the communication is still important.\nKN: Yes, communication is the key. And you as a Program Manager are on the “dark side” currently\nMR: Yes.\nKN: I have heard that developers called that.\nMR: But we have the cookies so that is nice.\nKN: Do you have some animals at home?\nMR: Not at the moment. We had some rabbits…\nKN: Oh, rabbits…?\nMR: yes, and chickens, but they were all eaten by the wild animals around us. We have eagles on one of the trees next to our house and we have visits of bears and coyotes and our neighbors sometimes see cougars and lynx.\nKN: So all those animals are in your area?\nMR: Yes, and some of the neighbors have dogs so we can sometimes use them as pets so to speak. So right now not. Chickens got eaten, rabbits got eaten so at that point we decided to give up.\nDW: But this is an unusual answer. Typically we hear about dogs and cats and sometimes about parrots …\nKN: … or fish …\nDW: … or hamsters but I have never heard about coyotes or bears.\nMR: I would not call them pets but they are living in our neighborhood.\nDW: My next question would be: how are you preparing when you have a conference like this or you giving lectures to user groups. How do you prepare yourself to do a speech?\nMR: Well, I have been giving presentations now for a long time so the preparation is prematurely along the following mentions. First if it is a new topic I obviously have to build up the material, understand the story I am going to tell and make sure that basically the slides and demos kind of help me tell that story. If I have given the presentation a few times at that point in time the main focus is to make sure that the demos run and that I am not having any surprises with let’s say the flow of the slides. Sometimes the PowerPoint seems to be losing my animation orders and stuff like that. But prematurely it’s like understanding it again, understanding the audience, what is the level of the audience, what is their interest and then tailoring even the previously given presentation. Tailoring it to that audience either by making the story telling more relevant to them or adding some demos that is more relevant to them. But one of the secrets of successful presentation is normally that you have a set of base stories that you can kind of combine into different things and then have different levels of your story telling. So if you talk to CEO type person you focus more on the business value on like understanding the high level picture versus if I am going and talk to the audience at SQL PASS then I am probable more talking about like coding experiences. Or if I go really deep I go and talk about like internal architecture so that people can better tune or performance tune or debug their performance of the product. So it all depends of the target audience. But the main thing is to understand the story you want to tell, make sure that your slides go logically into that story, bring people along on the story so that means you start a little bit high level and then you dive in and make it relevant to that audience. Like if you talk to business people – make it more business focus. If I talk to my field people then I need to sell that and make it more relevant for them so they know how to sell it. But if it’s for engineers make it relevant to the engineers so they can understand how to use it, how to optimize the performance without the debug and so on.\nKN: Do you still feel nervous before the presentation?\nMR: That depends on how well I slept the night before. Normally I am not felling like this anymore like I used to when I was like in early days of my presentation career. Unless it’s like in front of thousands of people. So the audience size matters a bit in terms of nervousness or if it’s like a keynote presentation where you are like supposed to give a keynote that’s probably more nervous than a topic that I understand deeply. It also depends like if I own the area of the or am very familiar with the area then I am less nervous than if it’s something that I have not presented to the audience or if it’s like not necessary my area of expertise.\nKN: Yes, makes sense.\nDW: I would like to go a little bit more to your background. When did you find SQL Server for the first time? Was it during your studies?\nMR: So I did do my PHD back in Zurich at ETH or Swiss Federal Institute of Technology for those that want to know the English name and during that time my professor Hans Schek, he was taking classes at Microsoft research back in the mid 90’s and he came back and talked to us about the OLEDB and SQL Server being re-architected using OLEDB to like make this composable infrastructure architecture in SQL. In which if you look at the SQL Server you probably would not notice if you looked from outside but internally it’s still like it uses for example the rowset provider like the connection between the storage engine and compute part of the engine. So that was kind of the first time that I have heard about it. The first time that I really got in contact with was when I was in Stanford doing my post doc and SQL Server was just launching SQL Server 7.0 and the research group led by Jim Grey and Microsoft in San Francisco basically had an event that I went to and I saw the SQL Server 7.0 for the first time. It had some interesting capabilities like linked servers and English query and I found it kind of interesting. Although it wasn’t really the reason I end up working for Microsoft I must say. At that time like I was building my own database systems in semi-structured data management project at Stanford. I was working on a project and before that I built up my own scale out update SQL database or object-oriented database on top of Oracle. Now one thing I knew was when I finished my PHD in Switzerland I had two career choices. I could become an Oracle consultant and make a lot of money helping people understanding the complexity of Oracle or I could go and build a better database system that could make the consultancy superfluous. So I basically decided to go and build something better than the complexity that was Oracle 7 at the time.\nDid that answer your question, Damian?\nDW: Yes, yes, of course. The next question would be, because you know we are, me and Kamil and other people, we run a community which is quite big in Poland and how would you convince people to be a community leader or at least a community speaker? Because we have a lot of great people but they are somehow afraid of talking in public.\nMR: Oh, well, in the thing that I think the community engagement helps you with is getting over that fear. I mean you are among people that are basically your peers. You have people that know the area, that seen you more as a friend. It’s not like in work where if you give a bad presentation you probably have a few people that will be cheering that it was bad, so then might get a promotion not you but in the community it’s also all about helping each other. So going to the community and then speak gives the ability to exercise your speaking muscles so to speak. Learn from it and basically get the confidence to then speak on eventually more important events as well. Also being in the community, I think, you often like one of the things that I feel the more I am helping on the community just being online in the community like, I don’t know, StackOverflow and events like SQL PAAS or the event where I was speaking last week with you guys. I feel the more I talk and contribute actually the more I learn. And the more I think I grow personally as well because people have feedback that is potentially unexpected could help me understand where they are coming from and applying that in my work as well. So I think contributing, participating actively is overall net positive. People are nice, they are encouraging you, they are here to help you with the feedback and so on. It’s not like they are there to kill you if you are pronouncing a word wrong or have to many pauses or ‘s in your speak. It’s all about basically making each other great, helping each other out and everybody has something that they can teach somebody else.\nKN: So they actually shouldn’t feel any fear.\nMR: No, they shouldn’t be afraid of that. I mean they can potentially just talk about cool code they have written and the thing that often helps is like if you present on something that you are really passion about then the passion shows through. I mean, if your demo is crashing or so on, people will forgive you that if they notice that you are really passionate about the thing. And you will learn. I mean it’s all learning experience. Nobody who is presenting on at the high level did that first time he presented. So I remember the first time in the high school I presented. I wanted to go to the floor and hide. If you see me now presenting you wouldn’t think that has ever happened before.\nKN: Tell us, are you perfectionist? In your work, in your private life?\nMR: I wouldn’t call myself to have OCD but I am definitely trying to have a like optimize the perfection to the level where you can deliver it. I mean obviously you have to sometimes deliver – deliver is a feature is what I normally say and which I learned from some of my early mentors at Microsoft. So being upmost perfectionist and never deliver is obviously wrong thing to do. However, I try to design the things in a way that makes it easy for people to use and kind of have a nice experience around it. I mentioned I drive car for fun and so one of the things that I notice is that some car manufactures are better at like optimizing the benefit cost tradeoff between giving you a good driving experience and still make the car affordable then others are.\nAnd you can kind of see if you have this then you are to some extent a perfectionist. But you can learn how to deliver something that is maybe not a 100% perfect in the first release but allows you to improve perfection up to the point where the return of investment becomes negligible. As an example, when we did the XML support in SQL Server 2005 one of the things I was never happy about was the performance and the size complexity of the XML indexing that we had at the time. It was good for certain use cases, it was very good for some like a generic use cases but it had a little bit of a high complexity in terms of optimization and it very quickly led to very huge indices and so on. I was kind of my perfectionistic things would have wanted to have better indexing story right from that but we needed to shape the things. So we basically left what we had and then we improved on it based on the customers feedback and learnings that we have. And we had some really like a helpful customer and interactions and then this helped us to deliver the XML indexing feature in 2012 that gave you the path based indexing and so on which is giving you kind of better experience. Which is a way much better experience at that moment of time. So sometimes you have to shape and then would I have done more in the later things – yes, but at some point in time you reach the point where you basically polishing something that is already looking perfect for most everybody else. Then you probably should stop and do something else.\nDW: It’s not our first interview where we can hear that the most important thing is to love to learn and love to share the knowledge. And it’s great that we can still hear the same from you that you have to learn a lot I think during your career. You have to be very active and sharing the knowledge is also very important I think because from that point of view you can also learn from the other people. And this is what we are doing all the time. But my last question would be – which achievement are you satisfied the most? I think you have a lot of them but what is the best achievement for you during your career?\nMR: You mean work wise? Work wise… I think the achievements that I am most proud of is when I basically go out with some capability and I see people using it and providing both their pleasure and sometimes the displeasure about the feature and how it helps them solve their product. I think that’s what I kind of feel is my, what I am proud of. Basically, that I have to deliver a lot of things to both SQL Server and now in the Azure Data Lake and so on. That really helps people solve their problems. That was the reason I didn’t go to the academia but decided to go to like a development and research to help solve peoples’ problems. So if I see people using my stuff to solve interesting problems that kind of gives me huge satisfaction.\nKN: I like the sentence you used during your presentation when you were talking about the version 1 of the Data Lake that people “love to hate”.\nMR: That was about Azure Data Factory you mean?\nKN: Yes, about the Azure Data Factory. I noticed that you are not a fan but you very frequently use the Twitter.\nMR: Yes.\nKN: Are you a fan of Twitter? Why do you use that way of communication?\nMR: So I mean I use Twitter prematurely as a way of getting information out. It’s very like a low friction way of communicating with a large amount of people and allows people then re-share it easily. Am I a fan of Twitter? For that purpose – yes. There are some other purposes, especially in the US right now that I think Twitter is a little bit over represented and so on. There is all like the Twitter bullying aspect and so on that has fortunately not reached out our area as much as some others. I think it’s just one of the tools that we have to get your message out and keep engagement, two ways engagements with the community and with people that you want to have interaction with. It definitely has been quite useful for sharing information very quickly. Also listening to what people are talking about in the areas that I am interested in currently. Basically, if you use Azure Data Lake, USQL with or without the dash in your tweets I will probably pick it up to see what are you saying and then if I have something to contribute then I will reply to it.\nKN: OK, and the really last question is about the team you are working with. How big the team is? And how many teams are?\nMR: So, I actually don’t know the actual number right now but basically on the one hand we have the PM team which is a not just me but my management and my colleagues that cover Azure Data Lake Storage and Analytics, different aspects, jobs, code experience in Visual Studio, development experience and the actual language experience. Some of the things we are still working on like interactive capabilities and so on. And then we have the engineering team. Engineering team I am prematurely working with is the language engineering team and the tools engineering team. So we have people for that obviously in Redmond and we have the tools team for example like Visual Studio and VS Code team in our Beijing office. We have some people that are working on some stuff in Serbia, more on the interactive side. The store side of this is in India so we are kind of wide geographically dispersed as well. And the interesting part of it is obviously that we are not just build the Azure Data Lake service we are also building and running and supporting the internal big data service called Cosmos inside Microsoft. And that means we are daily answering questions and helping with issues, etc. that our internal customers have in addition to build that external service. And that’s probably something that is kind of unique in that sense because let’s say if I was doing SQL Server I was having a box product. Or maybe an Azure SQL Database and I didn’t really have like an internal business running directly on the infrastructure that I was responsible for. I guess Microsoft IT would be using SQL Server but they have their own database administrators doing that. We are the people that run the internal big data platform for Microsoft as well. So that’s kind of where it is a big difference there.\nKN: The Microsoft employees are testers…\nMR: to some extents yes, as well, yes.\nKN: … are we talking about the team of 4 maybe 10 people?\nMR: So the PM team currently on the ADLA side, all up with everybody is probably about 30 people. 30-40 people. If you are looking at the engineering side, just the language side, I would assume it’s about a 100 people maybe depending whether you call counts of the tools team etc. So it’s definitely not as big as SQL Server.\nKN: But still is a big one.\nMR: yes, pretty big team. But you need that because like we have to keep our service running internally as well as do innovation on the external service and internal service as well, so you need a certain critical mass of engineers to be able to deal with that.\nKN: and also the product must be developed very fast.\nMR: Yes, right now we are trying to do a half yearly planning sprints and then basically try to shape about every 4 to 6 weeks refresh based on what we are doing on that. So yes, we have to be pretty quick. Now some of the features still take a long time like our partitioned output which is the frequently asked question on our feedback page after I think if you count all together the interactive parts. That feature was in development for a while but because it’s a fairly fundamental re-architecture of fundamental assumptions that we had earlier it’s going to take a while. And lot of components have to pull the work together. We are getting close to release it, hopefully early next year (this was recorded in November 2017), so yes, even if you have a 100 people at the end many of those are working on different parts. At the end you have one or two people working on one particular aspect so it’s still taking some whiles.\nDW: Yes, actually I was just thinking how big the investment is? Because I can imagine that 40 managers plus let’s say one team is around 100 people and there is more than one team and we see this from the perspective from one or two features on the portal. And we cannot see how much work it really took you to deliver this feature and this is really impressive.\nMR: I mean the point here is we obviously want to make it easy and making it easy means that we have, there is more cost to it for us because making it easy is actually harder than just putting something out and you have to do the final mile yourself.\nDW: That’s right. It’s easy for us to use it or for our customer to use it so you have to pay for it to be. You have to do a great development on the other side to make it easy. That’s great.\nKN: At the end of our conversation tell us where we can find you. Where the people can find you on your blog and information and knowledge…\nMR: so basically the best way to reach me obviously is either we have an email alias that I have created specifically to reach out for Azure Data Lake Analytics and USQL access which is just usql@microsoft.com\nKN: I told you Damian, I told you…\nMR: My Twitter handle is @MikeDoesBigData, quite simple, although it is a long twitter handle and otherwise I try to read the USQL tagged questions on the StackOverflow and the MSDN forum on the Azure Data Lake forum. So that’s kind of where I am currently hang out. Otherwise you might try to catch me at events like SQL PASS, Ignite and other places. But these are the kind of the main places. We do have a blog on MSDN, I have a personal blog there at MichaelRys is that. You can find me there. Or the Azure Data Lake blog is where we kind of hang out and post stuff as well. And if you have any like feature requests or questions or so feel free to reach me out so I try to answer all mails that we get.\nKN: Great.\nMR: Sometimes it takes a little bit trying to catch all emails.\nKN: Ok, thank you very much Michael this discussion.\nDW: Thank you.\nMR: Thanks for having me and I am hoped to see you either diving into the Lake or at least skiing on it using SQL Server.\nKN: Yes, for sure! Thank you very much.\nMR: Thanks, bye bye!\nUseful links: Michael\u0026rsquo;s Twitter: @MikeDoesBigData\nMichael Rys on MSDN\nMichael Rys on Channel9\nMSDN Forum: Azure Data Lake\n","date":"2018-03-16T18:17:50Z","image":"/uploads/2018/03/IMG_7916__ASF008_PASS_Summit_with_Michael_Rys.jpg","permalink":"/2018/03/asf-008-michael-rys-interview/","title":"ASF 008: Michael Rys interview"},{"content":"SET IMPLICIT_TRANSACTIONS ON Is One Hell of a Bad Idea\nHow dangerous IMPLICIT_TRANSACTIONS might be\nWhat is a Managed Instance (preview)?\nYaay! Microsoft has just introduced Managed Instance which allows you lift and shift entire on-premises SQL Server instance to the cloud with minimal application and database changes.\nGetting Started with DAX Studio\nAnother helpful manual, full of examples about DAX Studio.\nDeploying Power BI: Scaling from 5 to 5000\nEugene Meidinger [T] shows you why a deployment of Power BI reports maybe pretty difficult. See all options.\nPreparing for Exam 70-774 – Perform Cloud Data Science with Azure Machine Learning\nGinger Grant [T] reveals a few preparation tips and many useful links if you want to commence your journey towards Azure ML.\nBiml Case Study: Automate an Extract for Patrick LeBlanc’s K12 Solution\nPatrick LeBlanc [T|Y] has created nice Power BI solution for the US\u0026rsquo; education system and Andy Leonard [T] helped building an extract process in SSIS using his secret weapon. Guess what?\nMagic Quadrant for Analytics and Business Intelligence Platforms\nFull Gartner\u0026rsquo;s report. Microsoft is a leader again. Congratulations!\nUnlock Query Performance with SQL Data Warehouse using Graphical Execution Plans\nFind performance bottlenecks in your Azure SQL DW directly within the SSMS window by reading Graphical Execution Plan\nPower BI Desktop March Feature Summary\nThe latest update from Power BI team. My favorite feature is Report page tooltips (preview).\nIntroducing SQL Information Protection for Azure SQL Database and on-premises SQL Server!\nDiscover, classify and label sensitive data in your database environment. In this new feature (public preview) you don\u0026rsquo;t have to do this manually. Also, use SQL IP to generate reports from these metadata.\n[[PL] Będę na SQLDay 2018](http://blog.sqlgeek.pl/2018/03/02/bede-na-sqlday-2018/)\nDo you know that 50% sessions will be in English? Check out why Pawel Potasinski feels that it\u0026rsquo;s worth to be on SQLDay. Join him, join us.\n","date":"2018-03-11T08:00:58Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/03/last-week-reading-2018-03-11/","title":"Last week reading (2018-03-11)"},{"content":"Good day SQL folks! Thanks for visiting our blog again! It\u0026rsquo;s March already and we are looking for the Spring to come.\nColumnstore Indexes – part 120 (“Merge Replication 2016-2017”)\nIf you are a columnstore index fan or if you use it on your daily basis than all Niko\u0026rsquo;s posts are just a must-read. I would like to remind you that Niko will have all day workshop during the sqlday conference in Poland. His workshop is on 14th May and the conference is on 15th and 16th May.\nUse the Azure Cosmos DB Emulator for local development and testing\nFinally! We can have an emulator to play with Cosmos DB locally. Once you are done with local development and satisfied with the result than you can just switch to your Azure account very easily! Great thing.\nSQL queries for Azure Cosmos DB\nI know you know that you can query the Cosmos DB using the \u0026ldquo;TSQL like\u0026rdquo; language. That is not a new thing. In this article, you can read about how to do it plus there is a video explaining all that stuff. And of course, I share a link to the Cosmos DB playground where you can run queries without setting up the database at all\nCreate your first function in the Azure portal\nDon\u0026rsquo;t be afraid of Azure Functions. They are there waiting for you. You must be familiar with them. The sooner, the better. The blog post shows you how to start with them.\nVideo - 18 minutes. Jovan Popovic, Program Manager at Microsoft will explain do an introduction to the main features of Azure SQL Managed Instance\nCheck Chrissy LeMaires\u0026rsquo;s twitter (@cl) as she posts a ton of useful PowerShell scripts like this one. It automatically stops your XE Sessions at a specific time\nStart-DbaXESession -SqlInstance sql2017 -Session \u0026#39;Long Running Queries\u0026#39; -StopAt (Get-Date).AddMinutes(30) Cheers,\nDamian\n","date":"2018-03-05T11:21:43Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/03/last-week-reading-2018-03-04/","title":"Last week reading (2018-03-04)"},{"content":"It\u0026rsquo;s was very busy and pleasant week for me. Tenth SQLBits which has taken in London this year finished yesterday. I have been working as a volunteer (one of the ORANGE team) monitoring each day. Friday evening super party as always. Stay tuned as many great pictures will show soon.\nStairway to DAX and Power BI - Level 13: Simple Context Manipulation: Introducing the DAX All() Function\nQuite detailed explanation how All function works. With a good example.\nCreating Excel “Data Dump” Reports From Power BI\nIf you or your Client needs to analyze the data in old-way style rather than Power BI - check this Chris Webb\u0026rsquo;s post.\nSQL Server Replication for Temporal Tables\nHow to replicate temporal history table? There is a workaround.\nAnalysis of Spatial Data Using Cosmos DB\nUse CosmosDB in many ways. In this extremely easy example, you will see how to build such a solution using an alternative database with scale capabilities.\nAzure #CosmosDB Graph API now generally available\nNot only an announcement but also newly added support for Python and PHP application.\nAzure Cosmos DB Consistency Levels\nEveryone asks about it trying to understand how it works.\nData Virtualization vs. Data Movement\nDo you always need ETL? What the drawbacks are if you create visualization directly from a source?\nData Lake, Big Data, NoSQL – The Good, The Bad and The Ugly\nAn essay about how bad the terms like Big Data and NoSQL are nowadays.\nUnlock Query Performance with SQL Data Warehouse using Graphical Execution Plans\nYes, you see it well! Graphical plan for Azure Data Warehouse.\nReference Date Table in DAX and Power BI\nAlmost every single model in DWH or Power BI needs reference Data table. That one contains weeks, fiscal years and holidays for a specific country.\nHave a good week! Cheers,\nKamil\n","date":"2018-02-25T23:59:56Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/02/last-week-reading-2018-02-25/","title":"Last week reading (2018-02-25)"},{"content":"Good day SQL folks! Thanks for visiting our blog again! Have a great week!\nBob Ward\u0026rsquo;s demos github repo\nThis is not for reading per se but I would definitely visit the page is you can find there Bob Ward\u0026rsquo;s demos.\nAzure Data Lake Command-line Quickstart using Azure Cloud Shell\nShort blog post about how to use the Azure Data Lake command line tool. This is important if by any chance you use Mac or Linux machine. And for other reasons\u0026hellip;\nSQL Server table hints - WITH (NOLOCK) best practices\nI have put this article here on purpose. Even if you think that NOLOCK hint should never be allowed please read this. At least you ensure yourself in not using this hint anymore.\nMigrating to Azure SQL Database with Zero Downtime for Read-Only Workloads\nI do always read what Joe has to say. This time he takes you through an internally written service named MSAsset.\nWhat\u0026rsquo;s New in CosmosDB\nVideo - 10 minutes.\nScott Hanselman talks to Kirill Gavrylyuk to find out what’s new in Azure Cosmos DB, such as the Cassandra API for applications that are written for Apache Cassandra, updates to the Azure Table storage API, the Apache Spark Connector, the Graph API, partitioned collections, 99.999% (five 9s) SLA.\nAt the end of the post - please do not forget that the February release of the SQL Operations Studio is now available.\nCheers,\nDamian\n","date":"2018-02-19T04:07:37Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/02/last-week-reading-2018-02-18/","title":"Last week reading (2018-02-18)"},{"content":" Introduction Cathrine loves teaching and sharing knowledge. She works as a consultant, technical architect and developer, focusing on Data Warehouse and Business Intelligence projects. Her core skills are ETL, SSIS, Biml and T-SQL development, but she enjoys everything from programming to data visualization. Outside of work she\u0026rsquo;s active in the SQL Server and PASS communities as a Microsoft Data Platform MVP, BimlHero Certified Expert, author, speaker, blogger, organizer and chronic volunteer.\nThis talk has taken place during PASS Summit in Seattle, WA, on 2nd November 2017 (Thursday).\nDo you want to know how Cathrine has started her adventure in IT? How difficult is to be a woman in IT and how we can help in struggling with prejudices? Who is one of her personal guru? Finally, how is a correct Norwegian pronunciation of her surname?\nListen to this podcast to get to know how Cathrine became her journey to IT and how tough can be for women in the IT.\nTranscript Kamil Nowinski: Hi Cathrine.\nCathrine Wilhelmsen: Hello!\nKN: Thank you for accepting my invitation.\nThank you for inviting me.\nKN: Say something about yourself. Introduce yourself to the people who listen to us\nCW: Yeah, so my name is Cathrine Wilhelmsen, I am from Norway and if you are Norwegian my name is Cathrine Wilhelmsen (Norwegian pronunciation) but not a lot of people can pronounce that :)\nKN: Yes, that’s very hard.\nCW: I am a very active volunteer in the PASS Community, I have been working as the PASS Community Evangelist for the last year and I am coming on my last month there and then I will go back to just being a volunteer. After that I will be working on Data Warehousing and BI projects and very focused on Biml. That’s my big thing.\nKN: Yes, true. Recently your new book has been released.\nCW: It has, yes.It actually hasn’t been released yet. You can preorder it.\nIt is done but I think it will be released early December. So that is called ”The Biml Book”. Very difficult title there. It is a full book by Scott Currie [T], the creator of Biml, and Mr. Andy Leonard [T], one of my personal heroes who lead that book project, and where a bunch of other authors as well, where most of us are BimlHeroes who have been helping out writing that book. So very proud co-author of that book.\nDamian Widera: well, I would like to ask about something else because for sure the book will be a great present for many people including me and Kamil as we also work on BI projects as well. But I would like to ask you about something else. Because there are not so many women in the IT projects. How to convince women or young girls including my daughter to start to be a very good it professional?\nCW: I think part of that is encouraging them to get involved from an early age. This is a very complicated question, to be honest. You will get a thousand different replies to this. But what I would say is if you have small children it’s very common to buy girls things like, you know, dolls and teddy bears and things like that. But try and see if you can find some other things like puzzles or lab experiments or something like that and do it with girls as well. They are not just for boys.\nIf you start at an early age with things like that and get them interested in something that is super fun then that interest will probably, you know, gradually increase over time to a point where they get into something like IT or even other STEM-related areas like math, science and so on.\nWhat I would say is start early, start with doing the things you would do with boys. When it comes to tech and IT, it is very gender separated at an early age. The same thing, you know, a lot of clothes are pink for girls and blue for boys. Stop doing that. You know? Treat them the same way. And that goes the other way as well. Play with dolls with boys as well. It goes both ways.\nSo that is it and then after that when they get a little older there are lot of things like “Girls in Data” and “Tech Girls” all of these different initiatives where girls can get together and learn about – you know – playing with robots, learning to program and doing all of these things that might inspire them to get involved.\nDW: Well, that is great because this is exactly how we started and right now as my daughter is 14 actually she still has to fight at school that she can do better things on the computer in terms of programming than her colleagues. And sometimes the teachers say that “as you are the girl you should not play on the computer that much. It is for boys”. And this is very tough for a girl because she still has to convince also the teacher that well “I can also do that and I can be better than boys”.\nCW: Yes, and that is the other side of it where I would encourage especially men in the IT industry to work on changing those attitudes and changing the things that they convey. So if you hear someone say that, talk to them, especially man to man and say that “Hey I don’t think that this is the right attitude” because we need more women and we need more diversity in our business. And I feel like if that is coming from, you know, an older man – and in many cases it does – then get another man to tell him that “Hey, we might want to change this” and talk to him that way and try to change those attitudes.\nKN: That’s right. Still I think the problem is that the women need to unfortunately proof their knowledge more than the men.\nDW: all the time\nCW: Unfortunately, it is still the case. We do need to speak louder. We do need to fight harder to be heard even if we are equal, in the IT industry. And that’s something we need to work on and that is something that everyone needs to work on. So, if you see anything where someone is treated unfairly – whether it is a woman or a man – whatever –speak up! And start fighting those attitudes and those biases.\nKN: Yes, I always do.\nCW: Thank you.\nDW: Last week, because I was trying to write a blog post and I was with my daughter in Prague during the SQL Saturday and she was with me on the stage doing demos. And I told her about you Cathrine. And I gave her your example that woman like you can be like in the PASS organization or in terms of presenting stuff – you are one of the best and one of the most recognized person in our community and that is great. So at least she knows you and knows what she can do with that. And right now, she is thinking “How can I achieve that”. That is her next goal.\nCW: Yes, so thank you for that compliment. And there are so many great women in the SQL Server community. And I think that is maybe my last point on this thread here is that we need more women to be out there to be role models. And something that I try to do is not just\u0026hellip; I don’t want to just talk about being a woman in tech, I want to actually be a woman in tech. I want to create blog posts, I want to be out there speaking, standing up, being visible and actually providing – you know – being a role model for young women and girls who might see “Oh, hey, there is another short, blonde girl who can actually do this”. To be a kind of role model so they can see more people doing this. So I think that is important as well for our community to have more women speakers, presenters, bloggers - actually doing these things, and just showing that we can, and we can do it and we can do a good job.\nKN: OK. So let’s talk about the beginning. When did you start your adventure with IT and how? Do you remember that?\nCW: Oh my gosh. Well, it’s been a long story. For me, I have always been a geek. So, back when I was 9 or 10 is when I started learning how to build my own websites. And I hated the drag and drop interfaces because they could not do what I wanted them to do, so I started learning HTML and CSS and that is kind of how I got into this. A few years later I started learning MySQL because I needed the backend system, and PHP as well because that’s what I used back then, and that got me into the world of you know – programming and development. Then I had a couple of years in high school, or the Norwegian equivalent of high school, where I studied media and communication and I did a lot of graphic design and interactive design and so on. So I moved on through university where I did even more hopping back and forth. I started learning something called interactive design, which is where I learned Java programming for the first time, and I thought that was really cool. I learned more about databases and then I ended up finishing with a bachelor’s degree in Java programming.\n[Whisper] Don’t tell anyone [/Whisper].\nKN: Now it is very popular you know – CosmosDB, etc…\nCW: So I have like this very diverse background and then\u0026hellip; so funny thing is that I could not actually find a project on Java programming as my bachelor’s project, so that is when I learned SharePoint. I was assigned on a SharePoint project and I learned that. Then I got hired as SharePoint consultant in my first job but they didn’t have any SharePoint projects, so they said “We are going to put you on this Data Warehouse project”. And that’s how I got into Data Warehousing and BI. It’s been very back and forth, but I have a long background and I always knew that I wanted to get into IT. That I wanted to work with IT. I know programming, design and database development, so I can do a lot of fun things, and I love doing all of these things. So that was a bit of a background. But I accidentally ended up in Data Warehousing and I stayed there because I love it.\nKN: You also achieved the MVP award. It was two years ago I guess?\nCW: In 2015.\nKN: Yes, So you know it is a very big achievement even for a man and you did that.\nCW: Absolutely.\nKN: And you did that as a woman\nCW: Yes.\nKN: So that is good.\nCW: And that is something that I am very proud of. I am the only Data Platform MVP in Norway and I am the only female MVP in Norway. So that is a big achievement. And again, going back to actually being out there and being a role model, I am very honored that Microsoft saw my contributions to the community and gave me that award. It meant a lot to me.\nKN: I thing you do a lot of valuable things so it is totally deserved.\nDW: yes, that’s right\nCW: Thank you.\nDW: It is great that in one point of time you did a project in SQL Server or Data Platform because you are now on our Data Platform Community and not in the SharePoint Community so it is good for us. For the entire community I should say.\nDo you have tools you like the most during your daily work?\nCW: I do. Am I allowed to mention specific product names in this podcast?\nDW: yes, you can say whatever you want…\nCW: I am a huge fan of the Red Gate tools, and SQL Prompt is by far my favourite tool out there. SQL Prompt I use every single day for pretty much all the work I do in SQL, so that is the product that I use and that I will invest in on my own if my employer or company don’t want to pay for it. I would buy that on my own. So that is by far my favourite product. And the other side, obviously, like I mentioned in my introduction is Biml. Biml is also… Part of it is free and that changed my life when I started learning Biml. And there is also the licensed version of that. So those are kind of the two things I can’t live without in my work day.\nDW: As you are an MVP - what do you think about this program?\nCW: About the MVP program, I think… I absolutely love the program because it recognizes people who do a lot of things for the community. And they are focused on awarding people who help others in many different ways, either by speaking, blogging, answering things on forums. And I am so happy to see that Microsoft recognizes that community contributions are so important. I think that the flip side of that is that you don’t have to be a technical expert anymore to become an MVP and a lot of people still think that if you are an MVP you know everything. So that can be a little difficult when you have to say “I actually do not know anything about that topic at all”. It is also a little black box because there are no set steps to become an MVP, so a lot of people are confused about that but I think in general it is a great award that helps build up our community and encourages people to share their knowledge and help others.\nDW: great, I have a very similar thinking about this program as well. Maybe we should as Cathrine about the hobby?\nKN: Exactly – what is your hobby?\nCW: My hobby? Well, my work is my hobby. No… honestly like I mentioned earlier I have always been a geek, I have always enjoyed programming, developing, graphic design, doing all of those things. On my blog, that is kind of my little creative area, where I get to do some web development and graphic design as well. So I have always enjoyed those things. But I do… once in a while I actually do things outside of my computer as well. So I love hiking. I am from Norway and we go hiking a lot around - you know – mountains and water falls, and exploring the nature and forests and all of that. It is great way for me to disconnect and kind of – just relax and disconnect my brain and focus on just being in the present. So I love doing that and I also quite enjoy – well, it’s not computers, but I do love reading, watching movies, I am a sci-fi and fantasy fan… I think those are kind of my main hobbies and I enjoy travelling a lot. I have been fortunate enough to be able to travel to a lot of events all around the world. So I love travelling and seeing new places and trying new food as well. I am not a good cook but I like eating food.\nDW: But I have to say one think. You totally missed on event and by the way Kamil and I we are one of the organizers of the event and this is SQLDay in Wrocław. And I think it is a good time and should definitely visit us next year and give us a talk. Because people actually have been asking about you so are definitely on our list (of speakers) and I think it would be also good to visit Poland.\nCW: I completely agree with you. I have missed that event for several years and I have been sad when I have seen all the amazing pictures and tweets on that. So I absolutely want to join you guys and let’s try to make that work!\nDW: yes, for sure.\nKN: Tell us something about your work-life balance because it is very hard if you are working in the IT, it’s very hard to find the appropriate way…\nCW: Work-life balance… that is a tricky one, especially for me now being so involved in the community, where a lot of work and the rest of my life is blending together. I think what is important to me is I work a lot remotely so I work from home, I work from other places and I think it is important to have kind of a set schedule where you work certain hours and then when you are done working you get up and go somewhere else, like turn off your computer and go somewhere else. You know I am still on Twitter most of the time because all my friends are in the community, so there are a lot of things that are related to work. But don’t have work emails constantly on your phone. Don’t be available on Slack all the time. Be clear and set those boundaries for yourself and say that “I am available for work during these hours”. And if I choose to be on Twitter and have fun conversations with friends after that, that is my choice because this is kind of my hobby as well. But I think that the important thing is to get up and go do something else and go away from your work area even if you are working remotely.\nKN: So, you are working remotely currently but where are you living?\nCW: Well, I still live in Norway, born and raised and I still live there. By living there, I mean that I own an apartment where I still store my things…. Been travelling so much…\nKN: But in which city?\nCW: Oslo in Norway, but for the last couple of months I have been spending a lot of time Denver, Colorado. Beautiful mountains, gorgeous for hiking.\nKN: Like here on the Seattle. I saw a lot…\nCW: It is a little different… There is not a lot of water in Denver, but it is still gorgeous.\nKN: What are your daily habits?\nCW: My daily habits… Oh, my gosh…\nKN: Oh yes, hard question…\nCW: Yes, that is a difficult question.\nKN: I didn’t prepare you for that kind of questions…\nCW: No, you didn’t prepare me for that one. Daily habits. Well, I get up, I eat breakfast, I have coffee and then I start my work day.\nThe one thing I like to do is, for lunch breaks, to get up and go outside and go for a little walk or something like that, which helps me since I am sitting down so much during the day. And then you know have dinner, try as much as possible to go to the gym - and that does not happen every day - and then relax for a few hours. I’m not sure I have any other daily habits… I think that is it, yes, that is it.\nKN: Ok, that’s enough. You know I think you are a very good speaker …\nCW: Thank you.\nKN: …because you do a lot of speeches across many conferences basically and the question is how do you prepare yourself to do a speech?\nCW: Oh, I spend a lot of time preparing for my presentations. I probably spend on average around twenty-thirty hours on a new presentation building the slide deck, building the content. It is very important to me to have a good flow and a good story in my presentations. And then I also spend a lot of time just making sure that my slides look nice. They look nice to me, I’m sure others disagree… But just trying to make it very visual, finding some pictures, illustrations, working on that. And then I always – you know – I rehearse the session, I go through it a couple of times. I try to rehearse in front of the mirror, which is very awkward, but it is something that you have to do to prepare - I just run through it couple of times, make sure that I write down how much time each section of my session takes. And that actually allows me to see. I don’t have to follow a script but I know that I have 10 minutes to cover this topic, 10 minutes to cover that. And then if am early or late I can adjust it and kind of still stay on time.\nKN: And then you can feel comfortable…\nCW: Yes, I can and then I know if I need to speed up a demo or cut a demo if I am running late or if I need to explain something a little more detailed. And that seems to work well for me at least.\nKN: Do you stress before…\nCW: Yes, I do. I don’t get as nervous anymore but if it is a new talk I am always nervous because if I haven’t delivered it before… If it’s a talk I have done couple of times I am a little less stressed but I always get butterflies in my stomach and I am always a little nervous. And I think that’s good because I think it’s a good sign that you care about what you do.\nKN: Exactly, I think so. I think exactly the same. Tell us something about the PASS organization. What are you doing exactly in the PASS?\nCW: So in PASS I have been working as a community evangelist. I have been focusing on SQLSaturdays. SQLSaturdays, as you may know, that are free, one-day training events and we have around a hundred each year all around the world. And a large part of my job has been working with SQLSaturday organizers to help them publish new events and follow up with any questions, but also mentor new SQLSaturday organizers. Help them if they go “Hey I want to do this. How do I get started”. Explain the steps and what they need to do. Give them helpful tips on how to find a venue, sponsors, speakers, and marketing and all of these things. And then I am lucky enough that I have been able to travel to some of these events and be on-site representing PASS. Again, sharing what is PASS, what do we do, and talk to people about getting involved in the community.\nKN: Unfortunately, you are leaving the PASS organization if it’s not a secret…\nCW: It is not secret… It has been announced. I am leaving PASS in mid-December, so I have – when we are recording this –it’s about a month left. So I am getting through PASS Summit, which is where we are currently at and this week is a lot of fun, and the next couple of weeks I will be working on finishing off the projects I have and handing over to my co-workers who will be taking over the things that I have been doing. But I am not leaving PASS. I am leaving my role in PASS HQ, but I will still be involved in PASS as a volunteer.\nKN: Yes, absolutely. I can imagine.\nCW: Continue speaking, continue organizing and volunteering and, you know, even things like mentoring others. I have a lot of experience now talking to the members of the community. I am more than happy to continue helping in that way. But I do want to go back to tech. Like I said, I am a huge geek and I miss development and I really want to do more on Biml projects with all the cool things they [creators of Biml] are working on now. So going back to tech and doing that on a daily basis and then doing all the volunteering on weekends and on my spare time.\nKN: Are you missing the tech?\nCW: Oh, yes. I am missing developing things and seeing things actually being created.\nKN: Are you learning something during such conferences? As an attendee?\nCW: Yes, I learn a ton. I like to go to sessions about topics I know nothing about, to be honest. Things that I do not work on, introduction sessions to things that are so far away from what I do. I love doing that just to get an idea of all the possibilities that are out there. Recently I went to Microsoft Ignite and that’s a great place just to check out some of the sessions - you know – products I don’t use on a day-to-day basis. Just to see how this stuff ties into what I am doing. And the more you know about these things, I feel like the better prepared you are.\nKN: This is probably the hardest question in this interview. Are you a perfectionist?\nCW: Yes! Very difficult question. Yes, I am.\nDW: Difficult question and a simple answer. But from your perspective as a professional I know that you are because that is my feeling when you presenting. This is what I feel watching your sessions. But what about your daily life?\nCW: Very much the same. So, if you ever see the inside of my apartment things are very neatly organized and sorted and you know my clothes are – they are not messy at all. I am very much a perfectionist in my personal life as well. I am not sure we use perfectionist – but I like ordering things, I like things being aligned properly and I always make joke if you have a table cloth with checkered or any kind of pattern I have to put my glass down so it’s centered in the pattern. It is a horrible thing but it’s my personality. And that’s why I think working with data is so good for me because I am naturally very good at ordering, sorting and grouping and making sure that things are the right way. And same thing for my presentations – making sure that things look nice and attention to detail. Which in some ways can also be a curse because I have been working hard on learning that – you know – “80% might be good enough”. I want it to be 100% all of the time but it might not be practical to continue striving for that. I am also trying to learn that 80% is good enough. It doesn’t have to be completely perfect all the time.\nKN: Yes, exactly which is probably unachievable\nCW: Yes, it’s not achievable and it can stress you out if you always try and get to that perfection.\nKN: What hints would you give to young people who wanted to start in the IT market?\nCW: Oh, find something you enjoy doing. I think that’s the most important thing. I could say that “find the hot topic right now” and aim for that because you will get a good job or whatever, but I feel it’s more important to find something you enjoy, that you think is fun and that you really want just to sit down and learn more about because that is the kind of passion you want in a job. You have to do this every day. Find something you think is fun and dig into that and at the same time try and learn more about things that you might not use and just kind of dive into a topic that you enjoy and learn a little bit about everything else.\nKN: This PASS Summit conference, here are a lot of volunteers and I think this is a very good opportunity to tell the people why it’s worth to be a volunteer on the conferences and be engaged in the other organizations.\nCW: Absolutely, and that’s what PASS is all about, it is by and for the community, by and for the volunteers. There are so many ways that by volunteering you can advance your career and you can grow personally. So one thing is obviously by volunteering you are on-site, you get to talk to lot of the speakers if you are a room monitor or something like that. You can get to know more people and network in that way. But it’s also helping you with soft skills like communication and it is also good for your resume if for example you just have a single job on your resume if you are a young person in IT for example. If you have one job, but then if you add “hey I volunteered at my user group” or “I volunteered at a SQLSaturday” then you are showing potential new employers that you are active and you are willing to take that extra step. And for me if I had two people applying for a job, one who was good at their job but did nothing else and one who was good at the job but also volunteered, no doubt I would pick the person who did that little extra because it just shows that you are passionate about what you do.\nDW: How important is from your perspective a passion to do something. How can it help young people to achieve something that is really big because people would like to dream big? They should dream big but how the passion could drive them there. What is your opinion?\nCW: I think if you find something you are passionate about, it’s something you want to do, and it does not make it a chore. Something you want to do and keep doing that and if you enjoy what you are doing and it is something you want to do more of and that passion – especially in technology. You can learn more about a topic for example and get deep understandings of technologies if you are passionate about it. That can help you get better at your job or in terms of volunteering as well. The more you do it, it’s going to help you because you are building those connections, meeting people, you get to travel to events. I think if you have passion and you enjoy what you are doing then there is no limits and you can go and do whatever you want if you keep doing that.\nKN: At the end of our conversation tell us where can we find you – on twitter, facebook…\nCW: Well, I am easy to find as long as you can spell my name :) And I know I have that funny Scandinavian spelling. I blog on https://www.cathrinew.net/ or https://www.cathrinewilhelmsen.net but that’s longer to type. You can find me on twitter @cathrinew and if you go to either of those places you can find links to my other social media as well. I have done a couple of interviews that are part of my YouTube channel. I post pictures on my Instagram, I share things on my SlideShare and you can connect with me on LinkedIn as well. The only thing is my Facebook is the one thing I don’t use for any work-related things. That is mostly for friends and family to stay connected in that way. I don’t post a lot of interesting things there. Everywhere else you can find me…\nKN: Facebook is for your private life.\nCW: Yes.\nKN: Thank you very much Cathrine for this interview.\nDW: Thank you\nCW: Thank you guys for inviting me. I had a great time here.\nKN: Thank you very much.\nDW: And see you soon in Wrocław.\nCW: Yes!\nKN: Thank you, Damian.\nDW: thank you, have a great day!\nCW: Bye!\nUseful links: Cathrine\u0026rsquo;s BLOG\nCathrine\u0026rsquo;s Twitter: @cathrinewCathrine\u0026rsquo;s SlideShare\nCathrine\u0026rsquo;s LinkedIn profileCathrine\u0026rsquo;s as co-author of these books\n","date":"2018-02-14T06:10:47Z","image":"/uploads/2018/02/IMG_7881_ASF007_PASS_Summit_with_Cathrine_Wilhelmsen.jpg","permalink":"/2018/02/asf-007-cathrine-wilhelmsen-interview/","title":"ASF 007: Cathrine Wilhelmsen interview"},{"content":"Good day SQL folks! Thanks for visiting our blog again! I have just come back after almost 2 weeks of vacations and here you are - fresh, good content for the new week!\nAs it is early Monday morning - what could be better than a cup of coffee and great articles to read?\nHave a great week!\nAzure Cosmos DB Consistency\nThis article is all about the consistency on your Azure Cosmos DB. Take a look to learn how to set optimize the tradeoffs between consistency, availability, and performance.\nAzure Cosmos DB Advanced SQL Queries\nIn this 12 minutes video done by Andrew Liu you will learn how to tune query performance and issue queries programmatically.\nApache Spark Connector for Azure Cosmos DB\nAnother video, and again 12 minutes only. Scott Hanselman talks to Emili Lawton about Azure Cosmos Db Spark Connector which enables Azure Cosmos DB to act as input source or output sink for Apache Spark jobs. Cool.\nServerless CQRS in Azure\nThe first article of a series about Projection Processing, Querying Materialized Views and a comparison with an Azure Function approach.\nThe Annals of Hilariously Bad Code, Part 1: Critique the Code\nThe very bad code posted by Erik Darling. Spend some time analyzing this antipattern before you go to part 2. And make a guess where the code comes from :)\nThe Annals of Hilariously Bad Code, Part 2\nThe code form part one (above) is now clean and fixed :).\nColumnstore Indexes - part 119 (\u0026ldquo;In-Memory Columnstore Location\u0026rdquo;)\nI love Niko\u0026rsquo;s blog series about the columnstore indexes. It is a must-read for all dba\u0026rsquo;s and dev\u0026rsquo;s.\nBy the way - do not forget about our workshop we are going to deliver during the SQLDay!\nThe full agenda of the SQLDay will be announced at the beginning of March.\nCheers,\nDamian\n","date":"2018-02-12T07:32:30Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/02/last-week-reading-2018-02-11/","title":"Last week reading (2018-02-11)"},{"content":"Joe Caserta: Exploring Modern Data Platforms\nWhat is a key difference between data warehousing and modern data engineering on analytics platforms? Listen to one of the leading authorities on big data implementations.\nWhat sample data is available to use with Power BI?\nTake a tour of report examples in Power BI. All of them you can even download!\nFrom unstructured data to dashboard with Azure Data Factory and Azure Data Lake\nThe goal of the post is to present how to create a simple data pipeline that moves unstructured data from an existing cloud source into Azure and visualizes it with Power BI.\nCheat sheet for Azure SQL Data Warehouse\nHelpful tips and best practices for building your Azure SQL Data Warehouse solutions.\nAdaptive Query Processing – Batch Mode Adaptive Join\nAnother quick review of Batch Mode Adaptive Join introduced in SQL Server 2017.\nSQL Server Machine Learning Services – Part 3: Plotting Data with Python\nOne post of the series about Python on SQL Server board.\nScraping a website with 5 lines of R code\nShort \u0026amp; cool example in R how to gather data from a website\n","date":"2018-02-04T23:59:35Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/02/last-week-reading-2018-02-04/","title":"Last week reading (2018-02-04)"},{"content":"In my previous article, I have explained what does the SCD and described the most popular types of Slowly Changing Dimensions. I also mentioned that for one process, one table, you can specify more than one method. In this article, I would like to show Type 1 and Type 2 for a table, which is probably the most common conjunction.\nSCD Type 1 \u0026amp; 2 in one permanent house Let\u0026rsquo;s create an example tables: [stg].[CustomerPhone] (source) and [dim].[CustomerPhone] (destination).\nI\u0026rsquo;m planning to implement different behaviour for selected columns:\nTelephone - always UPDATE this column, but I\u0026rsquo;m not interested in keeping history of the values. It is SCD Type 1 thus. Address - INSERT new row once the value change in this column. Means keeping history, means SCD Type 2. StartDate, EndDate - we need these columns to provide point in time for SCD Type 2. ScdVersion - optional column. CREATE SCHEMA dim AUTHORIZATION dbo; GO CREATE SCHEMA stg AUTHORIZATION dbo; GO CREATE TABLE dim.CustomerPhone ( [Id]\tINT NOT NULL , [Name]\tVARCHAR(50) NULL , [Telephone] VARCHAR(50) NULL , --For Update in place (without history) = SCD Type 1 [Address]\tVARCHAR(50) NULL , --For INSERT, so keep history = SCD Type 2 [StartDate] DATETIME NOT NULL , --Date attribute for SCD Type 2 [EndDate]\tDATETIME NOT NULL , --Date attribute for SCD Type 2 [ScdVersion] INT NULL\t--version attribute for SCD Type 2 ); CREATE TABLE stg.CustomerPhone ( [Id]\tINT NOT NULL , [Name]\tVARCHAR(50) NULL , [Telephone] VARCHAR(50) NULL , [Address]\tVARCHAR(50) NULL , ); GO Processing SCD through MERGE statement To achieve the goal, I will use one of my favourite method - MERGE. This T-SQL statement was introduced to SQL Server 2008. In one command you can make either UPDATE, INSERT and DELETE. Considering that you can use it in conjunction with OUTPUT - makes the very powerful alternative for SSIS.\nNevertheless, MERGE does not belong to pleasant for writing. The syntax is complex and very easy to make a mistake. Not mention how to use it correctly to implement established logic. And then, SCD Merge Wizard comes with help. I will use this application to create full MERGE statement for above table.\nIn short, I would like to run the following script:\n--Prepare test TRUNCATE TABLE stg.CustomerPhone; TRUNCATE TABLE dim.CustomerPhone; GO --TEST 0: Insert first row INSERT INTO stg.CustomerPhone VALUES (1, \u0026#39;ABC\u0026#39;, \u0026#39;555-555\u0026#39;, \u0026#39;5TH AVE\u0026#39;); SELECT * FROM stg.CustomerPhone; EXEC dbo.MergeDimCustomer; SELECT * FROM dim.CustomerPhone; GO and whole logic should be applied. Only we need to have is Stored Procedure which contains \u0026ldquo;magic\u0026rdquo; MERGE. Let\u0026rsquo;s create the statement in an easy way. Using\u0026hellip;\nSCD Merge Wizard You can download this free, open-source application from GitHub. When you install and run it - you\u0026rsquo;ll see the main window:\nClick \u0026ldquo;Next\u0026rdquo; in order to go to the first step: set Source Connection. Use \u0026ldquo;Edit\u0026rdquo; button to establish the connection to source database:\nThen select a source table like below:\nNext step is Target Connection. If your server is the same as a source - just use \u0026ldquo;Copy from the Source Connection\u0026rdquo; option. Then, the only difference here will be a destination table.\nClick \u0026ldquo;Next\u0026rdquo; to go to \u0026ldquo;User Variables\u0026rdquo; step, which contains certain pre-defined values we will use later on.\nStep: Transformations Yeah! This is the chamber of the sorcerer. At the beginning, the app maps all columns by names. Your role is to set up the transformation rules.\nHere we prepare all magic, I mean\u0026hellip; business logic:\nEach MERGE must have a column key: set \u0026ldquo;Business Key\u0026rdquo; for column [Id] Set \u0026ldquo;SCD1\u0026rdquo; for columns [Name] and [Telephone] as we want to update these fields every time. Set \u0026ldquo;SCD2\u0026rdquo; for column [Address] as we want to create a new row in dimension table once the value change. Next window is Business Key Check. Merge is very sensitive to duplicate values for column(s) defined as \u0026ldquo;Business Key\u0026rdquo;. They must be unique in the source, otherwise, it leads to unexpected behaviour or error. It\u0026rsquo;s totally understandable. In this step, you can check your source data with only one click.\nStep: Options Options are in the next step where you can establish the behaviour of MERGE and other parts of the script. The description of each option you can find on the bottom part of the window. For our purposes set the options as presented below:\nLast step: Query Congrats! Your first MERGE statement using SCD Merge Wizard has been prepared:\nNow, copy the body (T-SQL) and enclose it into Stored Procedure:\nCREATE PROCEDURE dbo.MergeDimCustomer AS BEGIN -- ================================================== -- Slowly Changing Dimension script by SCD Merge Wizard -- Author: Miljan Radovic -- Official web site: https://github.com/SQLPlayer/SCD-Merge-Wizard/ -- Version: 4.2.1.0 -- Publish date: 08/11/2017 03:17:27 -- Script creation date: 28/01/2018 12:31:13 -- ================================================== -- ================================================== -- USER VARIABLES -- ================================================== DECLARE @CurrentDateTime datetime DECLARE @MinDateTime datetime DECLARE @MaxDateTime datetime SELECT @CurrentDateTime = cast(getdate() as datetime), @MinDateTime = cast(\u0026#39;1900-01-01\u0026#39; as datetime), @MaxDateTime = cast(\u0026#39;9999-12-31\u0026#39; as datetime) -- ================================================== -- SCD1 -- ================================================== MERGE [dim].[CustomerPhone] as [target] USING ( SELECT [Address], [Id], [Name], [Telephone] FROM [stg].[CustomerPhone] ) as [source] ON ( [source].[Id] = [target].[Id] ) WHEN MATCHED AND ( ([target].[EndDate] = @MaxDateTime OR ([target].[EndDate] IS NULL AND @MaxDateTime IS NULL)) ) AND ( ([source].[Name] \u0026lt;\u0026gt; [target].[Name] OR ([source].[Name] IS NULL AND [target].[Name] IS NOT NULL) OR ([source].[Name] IS NOT NULL AND [target].[Name] IS NULL)) OR ([source].[Telephone] \u0026lt;\u0026gt; [target].[Telephone] OR ([source].[Telephone] IS NULL AND [target].[Telephone] IS NOT NULL) OR ([source].[Telephone] IS NOT NULL AND [target].[Telephone] IS NULL)) ) AND ( ([source].[Address] = [target].[Address] OR ([source].[Address] IS NULL AND [target].[Address] IS NULL)) ) THEN UPDATE SET [target].[Name] = [source].[Name], [target].[ScdVersion] = [target].[ScdVersion] + 1, [target].[Telephone] = [source].[Telephone] ; -- ================================================== -- SCD2 -- ================================================== INSERT INTO [dim].[CustomerPhone] ( [Address], [EndDate], [Id], [Name], [ScdVersion], [StartDate], [Telephone] ) SELECT [Address], [EndDate], [Id], [Name], [ScdVersion], [StartDate], [Telephone] FROM ( MERGE [dim].[CustomerPhone] as [target] USING ( SELECT [Address], [Id], [Name], [Telephone] FROM [stg].[CustomerPhone] ) as [source] ON ( [source].[Id] = [target].[Id] ) WHEN NOT MATCHED BY TARGET THEN INSERT ( [Address], [EndDate], [Id], [Name], [ScdVersion], [StartDate], [Telephone] ) VALUES ( [Address], @MaxDateTime, [Id], [Name], 1, @MinDateTime, [Telephone] ) WHEN MATCHED AND ( ([EndDate] = @MaxDateTime OR ([EndDate] IS NULL AND @MaxDateTime IS NULL)) ) AND ( ([target].[Address] \u0026lt;\u0026gt; [source].[Address] OR ([target].[Address] IS NULL AND [source].[Address] IS NOT NULL) OR ([target].[Address] IS NOT NULL AND [source].[Address] IS NULL)) ) THEN UPDATE SET [EndDate] = @CurrentDateTime OUTPUT $Action as [MERGE_ACTION_91025df2-1344-4e4e-84f7-e790d1f77d7c], [source].[Address] AS [Address], @MaxDateTime AS [EndDate], [source].[Id] AS [Id], [source].[Name] AS [Name], INSERTED.[ScdVersion] + 1 AS [ScdVersion], @CurrentDateTime AS [StartDate], [source].[Telephone] AS [Telephone] ) MERGE_OUTPUT WHERE MERGE_OUTPUT.[MERGE_ACTION_91025df2-1344-4e4e-84f7-e790d1f77d7c] = \u0026#39;UPDATE\u0026#39; AND MERGE_OUTPUT.[Id] IS NOT NULL ; END GO Let\u0026rsquo;s get the party started! It\u0026rsquo;s high time to test our brand-new procedure. Let\u0026rsquo;s insert the first row into staging and check if the row will be copied to target dimension.\n--TEST 0: Insert first row TRUNCATE TABLE stg.CustomerPhone; INSERT INTO stg.CustomerPhone VALUES (1, \u0026#39;ABC\u0026#39;, \u0026#39;555-555\u0026#39;, \u0026#39;5TH AVE\u0026#39;); SELECT * FROM stg.CustomerPhone; EXEC dbo.MergeDimCustomer; SELECT * FROM dim.CustomerPhone; GO So far, so good.\nNew customer added to staging should also appear in the target table.\n--TEST 1: MATCH doesn\u0026#39;t happen using Id, then the record must be inserted INSERT INTO stg.CustomerPhone VALUES (2, \u0026#39;DEF\u0026#39;, \u0026#39;666\u0026#39;, \u0026#39;5TH AVE\u0026#39;); EXEC dbo.MergeDimCustomer; SELECT * FROM dim.CustomerPhone; GO New records always have min and max date, as we requested.\nTest: SCD Type 1 (Update in place) Now, we will test change for column [Telephone], which has been configured as SCD1. Update on this columns should cause a simple update on destination row, only for that column, nothing else. Let\u0026rsquo;s check out:\n--TEST 2: MATCH but only [Telephone] field has changed - UPDATE only --Means SCD1 UPDATE stg.CustomerPhone SET [Telephone] = \u0026#39;777\u0026#39; WHERE Id = 2; EXEC dbo.MergeDimCustomer; SELECT * FROM dim.CustomerPhone; GO Great. A new value for telephone can be seen, no other columns were changed.\nTest: SCD Type 2 (Update+Insert) Next test is to update [Address] column which has been configured as SCD2. That means we want to keep history changes and must create new row holding old version as well.\n--TEST 3: MATCH happens but only the [Address] field has changed I want to add a new record and update the start and end dates. --means SCD Type 2 UPDATE stg.CustomerPhone SET [Address] = \u0026#39;Summerhill Way\u0026#39; WHERE Id = 2; EXEC dbo.MergeDimCustomer; SELECT * FROM dim.CustomerPhone; GO In above example we experienced following actions:\nNew record has been created with [StartDate] as Current Time - GETDATE() The previous record has been marked as ended at Current Time (in normal scenario, I suggest to use real time of the change from OLTP, when available) Summary As you noticed - you can very quickly develop the business requirements doing the loading process into Data Warehouse\u0026rsquo;s dimension. Furthermore, you are able to save to a file whole mapping, connections and all options. Need to change the logic? Simply, open saved project, made the changes and replace procedure.\nAt any time you can review the Stored Procedure and even maintenance of that code is much easier comparing to SSIS.\nSCD Merge Wizard on GitHub - last release\nFacebook profile\nGive me a shout if you have any concerns or questions.\nThanks!\n","date":"2018-01-30T07:00:14Z","image":"/uploads/2018/01/gabriel-laroche-451471-fullHD.jpg","permalink":"/2018/01/scd-type-1-type-2-in-merge-statement/","title":"SCD type 1 \u0026 type 2 in MERGE statement"},{"content":"SQL Server 2017 CU3 adds CXCONSUMER waits, doesn’t mention it\nNew wait is not visible everywhere, but check where you can find it.\nSabbatical and the search for a dream job\nBasia Fusinska about her last few months of unemployment but busy time and plans how to be happy at work.\nMicrosoft SQL Server - Notes for Professionals\nVery helpful cheat sheet in book form. For those who always forget a syntax, like me.\nShortcut: When did my T-SQL query finish?\nQuick reminder about thing we use every day.\nGDPR - A guide for the perplexed\nGeneral Data Protection Regulation comes into full force soon. Have a look into reasonably clear summary and guidelines for implementing GDPR.\nWhat is database DevOps, and why is it important for DBAs?\nWelcome to DevOps for databases. Take effect the same principles as for application development: Source Control, Unit testing, Repeatable deployments and Continuous delivery.\nPASS Summit 2017 Video: “But It Worked in Development!”\n75 minutes of Brent Ozar\u0026rsquo;s session from PASS Summit 2017 where he shows symptom of problems during heavy load or concurrency.\nGetting started with machine learning in SQL Server\nSQL Server 2016 brings R language on board. 2017 brings Python. Start here if you want to get to know Machine Learning in SQL Server.\nStep by Step Guide to Migrate SQL Server Data to SQL Server 2017\nPlanning SQL Server migration? Create your own checklist, that one may help.\nHow I designed my VMware vSAN based Home Lab\nData center at home? Nothing easier - do it yourself. Like Klaus [T] did.\n","date":"2018-01-28T22:50:39Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/01/last-week-reading-2018-01-28/","title":"Last week reading (2018-01-28)"},{"content":"I\u0026rsquo;ve just realized that I did not publish pictures from SQLBits 2017 on the blog. It\u0026rsquo;s time to make up for it. Here you are - quick photo relation from the event in Telford. It was my third SQLBits and first as a volunteer.\nDo you join the conference this year? It\u0026rsquo;s coming\u0026hellip; SQLBits 2018: 21st - 24th February 2018, Olympia London. More information here.\n[srizonfbalbum id=8]\n","date":"2018-01-28T22:13:44Z","image":"/uploads/2018/01/SQLBits2017_Party.jpg","permalink":"/2018/01/sqlbits-2017/","title":"SQLBits 2017"},{"content":" Introduction Bob Ward is a Principal Architect for the Microsoft Data Group (Tiger Team) which owns the development and servicing for all SQL Server versions. Bob has worked for Microsoft for 24 years supporting and speaking on every version of SQL Server shipped from OS/2 1.1 to SQL Server 2016. He has worked in customer support as a principal escalation engineer and Chief Technology Officer (CTO) interacting with some of the largest SQL Server deployments in the world. Bob is a well-known speaker on SQL Server often presenting talks on internals and troubleshooting at events such as SQL PASS Summit, SQLBits, SQLIntersection, and Microsoft Ignite.\nThis talk has taken place during PASS Summit in Seattle, WA, on 2nd November 2017 (Thursday).\nDo you want to find out what is the next goal of Tiger Team for SQL Server?\nWhat was his first database he worked with and how SQL\u0026rsquo;s world looked like before DMV appeared? What does Bob have to do with the city of Łódź and why he does like to return to Poland?\nListen to this episode to find out all answers and more.\nTranscript Kamil Nowinski: Hi, Bob!\nBob Ward: Hello.\nKN: Thank you for talking the invitation for the podcast SQL Player.\nBW: Happy to be here.\nKN: It is good to see you again. We first met on the SQLDay about three years ago…?\nBW: Yes, I think it was quite a while ago. I enjoyed it very much. It was a very good conference.\nKN: However, PASS Summit conference is probably the best in the world.\nIt is certainly the largest in the world. I am not sure that I would say it is the best because one thing that I loved about smaller conferences like SQLDay is that they give you opportunity to meet people one on one and talk to them. I get to do that here at past. But it is like four thousand people, there is a lot of people here. It is a great conference. One of the thing that makes it really good is just that there are so many different verities of topics and so many experts. The top experts in the world of SQL Server topics are here.\nKN: There are many of paths here and a big opportunity to meet people and talk to them face to face.\nBW: Absolutely.\nKN: Could you please introduce yourself? I know that all the attendee knows you very well.\nBW: Sure, sure. My name is Bob Ward. I work for Microsoft and my title is Principal Architect. And now I am working for SQL engineering team. I have been in Microsoft for 24 years. I just turned twenty-fourth anniversary at Microsoft this last week. Most of my career up until last year was working at support. When I was in SQLDay I was still at support team. As an escalation engineer, I did a lot of deep troubleshooting on hard customer problems. Then I got to join the engineering team here and I am here a couple of years. I spend a lot of time still working with customers. I spend a lot of time as well doing evangelism, some of the work that we do. I do a lot of performance work like today. I did a talk on Linux yesterday. I was part of the keynote, as you probably saw.\nKN: Very internal things.\nBW: There is no question I have a reputation for doing that, right? And I love doing that. But I also have expanded it out to make sure that I do talks that are more intermediate level talks, explaning the value. The talk that I am doing today on performance is about SQL server 2017 and performance. I would not consider it deep internal stuff. I’ve kind of tried to broaden my expansion of talking about things that are not just internal.\nKN: Currently I heard that SQL server 2017 is the fastest in the world. What is the next goal?\nBW: I think so.\nWe will obviously continue to make it fast. Any time we release a new version of SQL server, we are always going to look for better performance. One thing to keep in mind about SQL server that I take pride in is price performance. It is not just about being the fastest database in the world, but it is about being the most affordable fastest database in the world. Maybe you saw in the keynote here in PASS, I did the demonstration using HP’s gen X server technology in the technology called persistent scalable memory. And this concept is where HP’s device has the ability to combine standard RAM modules – memory modules with SSD drives as storage. So, you can access memory as storage in very, very fast time. The cool thing about this is that it is 50% of a cost of a normal system for that the same amount space using older SSD’s drives. And if you think about that it is really groundbreaking. Think about a world where SQL server could potentially access memory as our disk in a way that bypasses of software storage layer. What if we could get rid of Page IO latch waits or something like that. I mean, think of those kinds of things, like get rid of all that IO waiting that you used to see in SQL. Those are the kind of advances that we still look out in the future.\nKN: It could not have been even imaginable ten year ago?\nBW: Yes, nobody would have thought we would be able to do such things. Think about drive and storage technology. That is one of the things that I love about our company that we are always going to be partnering with other companies, saying: “Let’s take advantage of the greatest innovation that is out there.” Though we are always thinking of price performance. It is not just about “fast”. It is like: “Can we find speed for people that is affordable for them instead of just spending millions of dollars on amazing hardware?” That is one thing I am pretty excited to look for.\nKN: Yes, so your job is really exciting.\nBW: I think so. I think it is an exciting product. If you think about the strategy of what you have seen us doing right now in Mircosoft SQL server, the SQL server engine still ends up being the most popular database engines in the planet but we are pretty serious about our Cloud Technology. Maybe you have heard about something called managed database instance which is us taking SQL Azure database and exposing that in the way that it looks like SQL server. You get the SQL Agent jobs, and link survey queries – things that you could not do before the Azure database were more constrained – now we have opened that up. But we still give you the power of the Cloud. So, I am actually excited about the ability to give our customers the choice. You want to run in your Cloud, we make SQL great for you. We will do it on Linux or Windows, we will do in containers. But if you want to choose our Public Cloud Technology, it is actually becoming pretty good.\nDamian Widera: Bob, do you think that at some point of time we are going to not used anything on premises. We go to the cloud only? Does it the only option for us?\nBW: Yeah, I do not see that happening anytime soon. We are pretty committed to giving you a choice. When we talk about Cloud, we often use the term Cloud first but not Cloud only for SQL server. What we mean by that is that we will do innovation in our Cloud products, iterate on it and make it successful and then bring it down to on-premise at the right time. You think about some of the work we are doing with automatic tuning or query stores or some of these features, those were first born in the Cloud and Usher database, and then we use them with some of our customers and got really good at it and then we push it is to our on-premise product. Damian, in the keynote you ask the people to show their hands of to see how many are ready to move to the Cloud. And very few people really raised their hand. It does not mean that people are not thinking about it, but we realized that a vast majority of customers still loves SQL server and wanna run it in their Cloud. So, you are not going to see anytime soon us abandoning that. We are going to continue to do that, but what we are going to do it and to give people the choice for those that do want to move to the Cloud and make that innovation and successful. One thing you are going to see us doing it to continue to expend on the scalability what is in the Cloud. We are going to start pushing for larger database sizes, larger memory footprints, faster storage. So that people who already made that transition, can do it. Some companies I talked to are thinking of the split model. With their current applications they are going to work on-premise but if they are building a brand-new application, they are taking a serious look at the Cloud as a possibility.\nKN: Yes, I think that people are still thinking about it and some DBAs are afraid of that.\nBW: I think DBAs are afraid of it only because they think like: “If this sink into the Cloud, I do not have to do anything more.” I think if you look at Azure database, that is false. Yes, we tried to put automation and things there to help developers and customers. But at the end of the day, query performance, query tuning in that design, some of the things are still important to make things run well. You are still going to have to need to do that even if you are running in the Cloud-type situation. Like I said, I think in foreseeable future customers are running in both worlds. Frankly, I think it is an exciting time for the DBA or data profession, because how about trying to learn both worlds? Having that skill set? That is actually a very good skill to have.\nDW: I would like to ask about SQL Server on Linux, because I can remember our discussion 3 years ago in Wroclaw during SQLDay and I know that we asked you: how big effort would it be to move from SQL Server to Linux and you said to us, well, it would be really big effort due to how SQL Server is built and now, 3 years later, we can see like a new-star: SQL Server on Linux works, it is very fast and people are really looking for an adoption. So, they try to do this. My question would be: do you think that you can go to any other operation systems? Can you see something like this apart from Windows and Linux?\nBW: That is a good question. I guess I was not a very good predictor three years ago, was I? [laugh] What I did not know back then but what I know now – quite frankly which happened not to long after I left your Poland conference - there was a Microsoft research project called Drawbridge. Its goal was to find the way to use SQL server in a Windows environment almost like a container. It was a really brilliant design and it provided the ability for teams like us to take the work they did and say: “I think we can find a way to make the core engine SQL server run on Linux but makes it thinking it is running on Windows.” But doing it in a way that is light-weight and fast. And you still get the performance. That is one of the things that we are able to make possible. When we have got SQL server on Linux we have pieces of code that are recompiled on Linux and these are those once that talked directly to the Linux kernel. But SQL server itself runs actually in the process based on Linux and that code is unchanged. It is still the same code that runs it on Windows. And when the engine runs, it does not really know. The reason we achieve that is this is still all Intel assembly code – no matter what happens, whether it is running on the Linux operating system or running on the Windows operating system it is still Intel assembly code. We have just done it in a clever way to run that Assembly code and not worry about the fact whether it is on Linux or Windows. Because – using this Drawbridge concept – we have this light-weight layer that knows how to talk to the Linux kernel and knows how to take all the calls that SQL makes that need the kernel whether it is IO or thread or memory make that call. So, your question about: “Can we do it on any other operating system?” We surely set ourselves up to that. It does not necessarily mean that is something that we have got on our plate, like we are looking at others OS footprints. But the architectural that we have built, what we called SQLPAL, which you should think of a new SQL OS that certainly has set ourselves up for it. If we want to that in the future, we probably could.\nKN: Bob, this podcast is not only about the technical stuff. Many people would like to know some of your habits, hobbies, etc. What is your hobby?\nBW: If you talk to me or see me here in the USA and when you hear me speak, you know that I am a big sports fan. Maybe you see me before I did demos, I used sports teams names. Absolutely sport is a huge hobby of mine. I play sports. I like playing golf. I have two sons: Ryan and Troy. My younger son Ryan is still in college and he plays college baseball in America. And I love watching him playing baseball. My older son Troy also played baseball in college. He is now graduated. I am a big sports fan. This is still something that I enjoy doing. I love all sports by the way. You might find it interesting, because you are from Poland, I was a soccer player in college. That was my sport. I was actually really good at it. So, I played soccer in college and I enjoyed it very much.\nThe other thing that I am really into is my Christian faith. I spend a lot of time at my Church – volunteering and helping out. I helped with our Youth program for many years, we called it the high-schoolage. I still do that, so that’s another big thing, that’s a huge part of my life\nKN: You told me yesterday that your wife has been in Łódź in Poland.\nBW: Can you believe that? You came up to me yesterday and it was so funny that Poland was on my mind. My wife had an opportunity to go to Łódź as a missionary to spend two weeks there in the program called Lets start talking where she taught English. We use the Christian Bible to do this with. She loved that city. She had so much fun in Łódź. She just said that everybody loved beets there and everything that she ate there had a lot of it. She also got to do some fun. She went to Warsaw for a few days. I am a big fan of history, especially the World War II history. And she got to go to some places in Łódź which I did not realize was one of the largest population of Jewish people during that time. And then she went to Warsaw to visit some historical places. I was kind of jealous of her. But she was in Poland for about two and a half week. She loved the Polish people. She loved them so much that one of the people that she met, who works as a missionary in Łódź, came to our home last week. He talked more about Poland and the history. I told him that I have been to Wrocław.\nKN: What about the work-life balance?\nBW: It is a struggle. Quite frankly I think it is a struggle for many people in technology. I have been in Microsoft for 24 years and it has always been a struggle. But I will tell you that my family is very important to me. So my family comes first. I do some creative things at Microsoft which is such an amazing company to work for. They allow flexible hours, flexible work schedules. Sometimes I work from home. So I tried to use creative ways to make sure that I dedicate the time to those hobbies I talked about but also to my family. It is a little easier for me today because both of my boys are grown and out of my house. But I still want to connect with them. And not only them, but also my mother, and my family from Texas. I just try to make sure that I am conscious about it every week. Sometimes I slip like everyone else and I am working at home. But maybe after a couple of days I will shut my computer off and I won’t turn it back again when I am at home. Or sometimes I turn my phone off for a couple of days. It is hard. But I tried to force myself to do it. One thing that I do on my personal vacation is shutting my email off for the whole time. I could be on vacation for a week with my family and I will not read any emails from my company for week. It does not matter. And I tell my boss: “I guess you can call me if there is an emergency like the world is coming to an end.” But I try to make sure I do not do that. And that is really helpful. That is a really good tip for everybody. Do not read your company email when you are on your vacation. It forces you to dedicate your time for vacation.\nKN: Yes, and you started to be addicted to checking emails.\nBW: Of, course. No question.\nDW: When did you start interesting in SQL Server?\nKN: This is an interesting story. I came out of college and develop the skill in C programming when that was a new thing back in the nineties and late eighties. I came out of college in 1986 and nobody now C programming back then. And I quickly learned to write C programming on UNIX systems. I joined the job when somebody said: “We’d like you to be our developer and database administrator on Ingres which was a database system many years ago and eventually Oracle. I just develop to learning for the SQL language and database systems because I was kind of force to do it in the job as a programmer. So in the early nineties, I was looking for a new thing to do. A friend of my wife was working for a Microsoft and said they were looking for people who new database products, especially Oracle skills. Because SQL server back then was just a very small product. Nobody really used or new SQL a lot. It was mostly Sybase so it would have been a plus. But I knew Oracle very well so one of the reasons I was hired at Microsoft was because I had this Oracle skills but I also had some of the development skills. The people who hired me said: “You could be very valuable in the support space because you know how to program and write code. You could probably debug easily customers problems.” So, I actually learnt very early on how to read the SQL server source code. That was fun for me. I was reading the SQL server engines code back in 1994 how we wrote it and thought these years one of the things I love about my job is that I have the ability to still look at our source code and kind of translate it for everybody. When you think of the internal talks that I do, that is really me translating our source code.\nKN: Yes, I still remember your workshop. I think that most of the people do not understand the whole session – including me!\nBW: It is hard. I have a fun in doing it though. I enjoy transferring knowledge to people. That is one of the thing I love about my job. I get a great pleasure of translating very difficult topics, not that everybody else are not smart, but you do not have an access to the source code either you have spent all your time on it. But that is kind of my history on SQL, I was kind of forced to do it at my job as a young college graduate. I loved it. This kind of database is fun. I had also some friends that said it was a great skill to have. Ingres, Oracle that was my background, now I’m a SQL Server guy\nDW: I just wanted to say that, because I’ve attended a lot of your sessions and I can see that you can explain everything, even if those things are very complicated. Because you have an enormous passion – so you can talk about complicated things in a very easy way that most people can understand once we have a teacher like you. That’s my opinion.\nThat’s so kind of you to say. Thank you.\nDW: I know that Kamil asked a lot of questions, but I would like to know if you have any animals at home?\nBW: Yes, I have two dogs. I have a Border collie and a Labrador. These dogs are now quite old. We have those dogs since my kids were young. The Border collie is like fourteen years old, and the Lab is like twelve years old. I still love them. I love dogs. Man’s best friend. Very loyal animals. I have some property – everything is bigger in Texas – I have three acres of land, so my dogs loved to run around my house. That is actually one of my hobbies – maintaining my property.\nKN: Damian maintained also that you can explain very easily the very complicated stuff. How are you preparing yourself for a speech?\nBW: That is a great question. Maybe you have heard me speaking and it seems that it comes easily. But it does not. I spend a very long time in a very hard preparation for every session I do. The way I usually build my presentation it is like anything you will do in life to build a paper or a book. So I outline my idea – what I am going to do. I use PowerPoint to do that. Then I start filling it in with what I need to do. And I always overbuild my presentation. Sometimes in my presentation, you can see me having some bonus material section – at the very back there will be extra slides that did not go over. I build more slides than I need and then based on the time I am allowed for my talk I take slides that I am not gonna cover and put them in the back.\nAnd another thing I do, this is gonna sound crazy, I like to exercise – running or lifting weights – and usually when I do that and I am having presentation coming up, I am thinking and rehearsing during those sessions. Like, I run for three miles, and during this run I will be rehearing in my mind what I am going to say and typically every time I run I have some new ideas. “Oh, I need to change this slide. I need to make sure I cover this.” Sometime when I am on my run I will take my phone out and I am recording on my phone: “Hey, Bob, don’t forget to change something.” I found out that exercising is a great thing to clear your mind and it helps you think about things in a new way. A lot of my preparation goes into these talks. I also am a great believer in making sure that demos are done early. Always do your demos early on, because you do not want to put you on your slides things that you cannot backup with a demo.\nDW: How do you teach something new. What about teaching new stuff, when you know that you go, for example, we have to learn a lot of Linux right now to be an expert. How do you do this, how do you manage that?\nBW: Sure. That is not easy. First of all, you mentioned passion. So I have a passion for the technology. That makes it a lot easier. If you are passionate about something, it is a lot easier to learn something new. You also got to love learning. If you are afraid to learn new things, transition your career, you cannot teach people stuff. Just having a passion is the first place.\nOne of the biggest things I do when I try to take on something brand new is to learn how to use it from the perspective of a user. For example, about a year ago I was challenged by my vice president to start teaching people about machine learning services. This is R and Python so far. I did not know a thing about that stuff. So, I started from a perspective of a user. I learned how to use the software first. Why would somebody want to use this? I looked for key studies, value propositions and business cases. So even though you see my talks that are very technical, I actually start to learn something new from the perspective of the question “Why? Why would you want to do this?” Because if you do not have that on your mind who cares about the technical part? So, I always start from this kind of high-level perspective and then I start to figure out: “How does it work?” Because as you have seen in my talks, I love to explain how it works. But if you cannot explain the value why would someone want to use that new technology, it does not really matter if you can explain how it works. So I kind of combine those two together. And that is how I teach something new, to start from that perspective: “How to do it? How to use it and why does it matter?” And sometimes I have been asked to look at the new topic that I did not really think was all that great. I did not have a lot of passion for doing this. Let’s take example on Linux. I used to be a UNIX developer, so I loved Linux. It was very easy to start building training on Linux because I like it. And I think it is a value.\nKN: Now, I can see. It was your idea. [laugh]\nBW: No. It was not my idea That’s why I brought up Tobias, Slava, and Travis, all those are the founders of it. But I think Linux is very valuable for customers that want to be on Linux. Three are people in the technology world that think that Linux is a better operating system than Windows. I am gonna stay neutral because I am supporting SQL server both on Windows and Linux. I think Windows has many great capabilities and that what I have used for years but there are people out there who are Linux people. That is what they want to do. Maybe you have used Postgres on Linux for years and that is what you liked to do, but Postgres does not meet your performance needs anymore because you scaled up. But you want to stay on Linux. That is why we did this. Put SQL in this platform so that people who love that operating system have an opportunity. We had a customer DV01 which we were showing out into the world that he is a great customer and that is exactly what happened to him: big Linux shop, big Postgres developers shop, the Postgres would not meet their performance needs so they did not know what to do. They saw us starting to go for the Linux proposition and they become big fans very quickly. That is how I believe about the teaching new things. You have to have passion for it and you have to be able to explain people why do they even want to use the technology.\nKN: Yeah, the business needs first and the user experience as well.\nBW: Yes, and then you start talking how it works. Because I think that if you talk about something how it works it gives people an appreciation. Machine learning services – one of the reasons I started presenting on how that works – is to give DBAs a trust factor that they can trust running this software. Because the DBA is like: “I’m not gonna let R script run in my SQL Server! I don’t trust that.” But if you understand the architecture and what we built then you become more comfortable with that.\nDW: About achievements – what achievement are you satisfied the most?\nBW: I tell you what. As for as SQL server goes. I think the biggest achievement I am most proud of is that there are features today in the product that I felt directly responsible for. Especially when I was back in support. You look at extended events, maybe you have seen this feature, I was one of the people that first proposed that technology along the Slava Oks to be in the product. Slava came to me and said: “Hey, we have this new technology. What do you think?” I was like: “Yeah!” I remember being in the meeting with the development team and the development leaders pushing the say: “We should put this in the product as a set good.” You look at dynamic management views that was also another one back in the days before we had DMVs, myself, Robert Dorr, we would do these really long debugging sessions to solve problems with Windows debuggers, which was great but it would take hours. So, Slava came to us and say to work together on this DMVs concept.\nKN: It was before the DMV happened?\nBW: We would use debuggers and now there is the DMV. Slava came back in 2005 when was the first release and she said: “Hey, let’s put the DMVs in here, they can really help the debugging using SQL queries.” It was not only for me but also for everybody else. I think also about the work I did with Paul Randall. He used to be in the development team. His big passion about corruption scenarios, check DB, things in the storage engine – all these different types of technology. We would work directly with Paul to add these features into the products. Those are huge accomplishment for SQL I am proud of.\nFrom my personal perspective, I am just excited that I have two great wonderful young boys. That now I consider to be great young men. They made of high character. It just swells me with great pride when I have heard other people telling me that they have asked them what they like about their dad the most. And they say: “My dad is a person to have high ethics, high integrity and high responsibility.” And when I hear that, that makes me feel very special. And it makes me feel special that I have been able to instil that same sense of character into them.\nKN: Do you think Microsoft Ignite that recently has happened is bigger or better conference than PASS Summit?\nBW: I have been in all three of them because I have been in PASS Summit since 2003. These are really two different conferences in a way. And I actually treat them a little bit differently. Microsoft Ignite is now the largest IT conference in the world. There are like thirty thousand people coming to that conference. And certainly, SQL server and our data technology is our part of that conference but it is much larger than that. It is a showcase in a way for people across the world that come to talk to Microsoft and our partners and the community about the wide range of Microsoft products, whether it is Cloud, SQL or Windows. The PASS Summit is all about data. This is a data conference. If you like data you come there and I think they complement each other. I think that you will see some content at Ignite presenting also at PASS like I am doing that because some people did not get to go on Ignite. But you will never see me do insight Linux at Ignite because the audience that usually goes there does not want to see that. A little too deep. Ignite will never be a deep internal conference ever, it will be about the value proposition or performance. Whereas PASS provides us unique opportunity. Right now, we are sitting in the data Clinic. We do not have such a thing in Ignite.\nKN: Yes, this is an amazing place. You can come here and ask about anything.\nBW: If you look at this right now, you can see all these people are asking these questions to these experts. Funny story about the Clinic, you will find interesting, maybe you have heard about this gentleman Ken Henderson – that was an expert on the SQL community many years ago. He passed away around 2008. But Ken was the reason that I got involved in all those speaking arrangements and in PASS. In 2003 at the first PASS Summit that I attended Ken and I started this Clinic concept by sitting in a room with a couple computers, a desk and a couple of chairs and we announced: “Anybody wants to ask us questions, come in.” It was not organized with all those desks and chairs and boards. It was just Ken and I – just two people. And the first day we had like two people. The next day they made an announcement at the keynote: Ken Henderson and Bob Ward are answering questions. The next day so many people came in, the fire marshal had to move people out of the room because there were too many people in the room causing a possible fire hazard. We called it something else back then. But now it is born into this IC of the Microsoft data Clinic which is amazing. Anybody in this conference, for no charge, can walk in here and talk to the top experts of SQL server at Microsoft in the world. Where else can you get that? You will get it at Ignite and here.\nKN: How can we adopt at least this small part of that Microsoft Clinic in SQLDay in Poland?\nBW: I think it is not hard at all. I remember being at SQLDay and there were several people from Microsoft there. Or even just the experts. If you think about your speakers in SQLDay – those are you experts. All you need for a clinic is a couple of tables and some laptops and a whiteboard.\nKN: I am talking about the people from Microsoft.\nBW: I understand. We organize these smaller Clinic events in other SQL Saturday events. We have got some of the local Microsoft resources – the PFE engineers, the architects of Windows all over the world. And we just say: “Let’s just get some small table set together and we will advertise a clinic and we will just help people come and ask us questions.” It is really easy to put this together as long as you are able to get people who are willing to staff it and answer the questions. You just need laptops to show something, couple of tables and you got a clinic. That is how we started that and look what has it become?\nDW: Are you a perfectionist?\nBW: Oh my Gosh! If you ask my wife about this, she would probably say yes. If you ask people who see my present and talk to me, they would probably say: “Yes”. I do not consider myself that. I do consider myself to be thorough. I am a big believer in accuracy and making sure that I do the best I possibly can to present things which are correct. I would rather do things that are correct and that you will understand them than them being polished. Course I have spoken now for 20+ years so I have gotten all of those kind of combined together now. But I do not think that I am a perfectionist. Maybe people think that I am, but I try not to focus too much on perfection. Yesterday I had a small incident at my Linux talk where I had a problem in my demos. And I was not happy about it but mostly people said that they could not see that I was angry. I am still not happy but I am not gonna let it affect the rest of life because of it. I would say I am close to the perfectionist but I am not that.\nKN: Bob, thank you very much.\nBW: Thank you for having me. I hope you find it helpful.\nKN: Yes, absolutely.\nBW: Make sure you send it to me so I can hear it.\nKN: Yeah, I will. Absolutely.\nDM: By the way, we would like to invite you, if you have time, if you would like to come, you’re always welcomed in SQLDay in Wroclaw.\nBW: That’s so nice of you. Thank you, you’re so kind. Thank you for an invitation, I appreciate it. Thank you for your time.\nDM: Thank you\nBW: Thank you Damian.\nFind Bob Ward: Bob\u0026rsquo;s blog posts: SQL Server According to Bob\nBob Ward\u0026rsquo;s Twitter: @bobwardms\n","date":"2018-01-24T05:06:30Z","image":"/uploads/2018/01/IMG_7881_ASF006_PASS_Summit_with_Bob_Ward.jpg","permalink":"/2018/01/asf-006-bob-ward-interview/","title":"ASF 006: Bob Ward interview"},{"content":"Good day SQL folks! Thanks for visiting our blog again! Here you can find some great articles I have found for you.\nHere you are - this week readings.\nStart reading on Monday morning :) Have a great week!\nAzure Data Factory V2 Developer Interface Launched\nPaul Andrew will go you through the new interface of the ADF v2. This is important because as you know the ADF can run the SSIS packages in the cloud.\nPASS Summit 2017 Video: \u0026ldquo;But it worked in Development\u0026rdquo;\nBrent Ozar has uploaded his session recording to the YT! Go and watch it. By the way, Brent also invited Pinal Dave to be on stage with him! Check out this post because Brent also posted tons of materials there.\nUsing Windows stored credentials to connect to SQL in containers\nAndrew Pruski is one of my favourite speakers / bloggers about containers. He has done a great job during the past SQL Saturdays and I think it is worth to mention his blog here. Read not just that one blog entry but try to read something else as well.\nInstall and Configure SQL Server 2017 Availability Groups on Linux - part 3\nEdwin Sarmiento describes how to configure the AG on Linux. Well you might think how this is actually possible as the first implementation of the AG requires a Windows Cluster\u0026hellip;\nDealing with Disappointment and learning from Failures\nMy absolute number one blog post this week. Cathrine has the courage to show her feelings about what could go wrong during a conference and how she deals with this. The blog entry is very personal\u0026hellip; definitely, this is a must-read for all of you this week!\nStay tuned - Cathrine will be one of our guests in our ASF podcast!\nBy the way - do not forget about our workshop we are going to deliver during the SQLDay! Join us there!\nCheers,\nDamian\n","date":"2018-01-22T02:01:31Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/01/last-week-reading-2018-01-21/","title":"Last week reading (2018-01-21)"},{"content":"Good news!\nWe (Kamil and I ) are going to have a full day workshop during the upcoming SQLDay 2018 in Wrocław. The workshop is dedicated to DevOps in SQL Server. We are going to cover topics like:\ndatabase projects - create, manage, deployment, store data in projects, what should be a part of the project and what should not be handling CLR objects migrate projects to Microsoft Azure use hybrid approach tSQLt testing handle known issues many more\u0026hellip; You can join us and spend the entire day exchanging knowledge and having fun with coding. The workshop agenda (in Polish) can be found here.\nIt is worth to mention that the SQLDayis the largest conference focused on Microsoft Data Platform – databases, Big Data, Business Intelligence and advanced data analysis. We are part of the organization committee. Please bear in mind that this conference is organized by the DataCommunity.PL - Data Platform community.\nCheers\nDamian\n","date":"2018-01-16T22:05:59Z","image":"/uploads/2018/01/DevOps.jpg","permalink":"/2018/01/devops-workshop-during-the-sqlday-2018-sqlplayer-team/","title":"DevOps workshop during the SQLDay 2018 - SQLPlayer Team"},{"content":"The top 7 business benefits of establishing DevOps in your organization\nNo comments, find and understand the benefits.\nColumnstore Indexes – part 116 (“Partitioning Specifics”)\nWhat do you need to remember while migrating partitioned DWH tables into Columnstore Indexes? Check it out.\nCritical SQL Server Patches for Meltdown and Spectre\nSteve Jones\u0026rsquo; [T] summarization of the latest worldwide processors issues, mainly from SQL Server perspective.\nUpgrading SSIS projects, part I\nAre you going to upgrade your SSIS projects to the latest version? Read it before.\nAzure SSIS – How to Setup, Deploy, Execute \u0026amp; Schedule Packages\nPaul Andrew [T] is going through the Integration Runtime in Azure Data Factory v2 - explore SSIS as PaaS.\nAzure Analysis Services Scale Out \u0026amp; Diagnostics (video)\nChristian Wade [T] with Scott Hanselman [T] explain how the Scale Out in Azure Analysis Services works.\nRedis vs. MongoDB: Comparing In-Memory Databases with Percona Memory Engine\nInteresting comparison between two most popular, NoSQL databases engines.\n8 Machine Learning Algorithms explained in Human language\nIf you starting in ML - those explanations might be more eatable.\n","date":"2018-01-15T02:12:32Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/01/last-week-reading-2018-01-14/","title":"Last week reading (2018-01-14)"},{"content":" Introduction Marcin Szeliga - Data Philosopher. Since 2006 invariably awarded Microsoft Most Valuable Professional title in the Data Platform category. A speaker at numerous conferences across Europe, as well as at user groups meetings. The author of many books and articles devoted to Microsoft Data Platform. An independent consultant, database-centric application practitioner and architect totally focused on SQL Server and Azure. Huge fan of Data Science and Machine Learning.\nListen to the story of the most experienced Polish MVP. Marcin explains his point of view on work-life balance and what is the future\u0026hellip;\nTranscript Hi Marcin, thank you for accepting my invitation. What is your name and where are you living?\nWell, so, I am Marcin. I am living in the Upper Silesia, the best part of Poland.\nWhat do you do for a living?\nHa! Tough question! From the very beginning. I am an independent consultant working with data, and Microsoft Data Platform for the last twenty years, mostly with SQL Server and Azure.\nAnd how long or from since have you been working with SQL Server?\nSince SQL Server 7.0. So 1998, something like that\nDo you have your favorite tools you use during your work?\nWooow.\nApart from the Management Studio. Something which is really cool.\nYes. Actually, it depends on the task. Quite recently I have been arranged to fix some performance issues, troubleshooting, so for this, there are all those great scripts from the community – from Adam Machanic, Brent Ozar, Glenn Berry – stuff like that.\nIf you are asking about tools – I am using the best tools I can find to solve my tasks. Like those scripts. When I am working with the data analytics there are probably the R libraries.\nSo you are basically tools, sorry, the scripting guy. You like scripts the most.\nWell well, what do you think about “the most”.\nThe most. So the graphical tools are not fine for you and you prefer to write scripts because you can probably automate them mostly.\nYes, that is the great “pro” of using that kind of tools because at the beginning you spend more time or you have to be lucky and find the tool that is ready to be used. But afterward, it is a piece of cake. Always pleasure. So yes, let’s say I am a scripting guy.\nAnd you must be using a lot of the R Studio\nYes, that’s true. I have started working with R just a few years ago and actually, the best part of R are those 10 thousand plus libraries ready to be used. R is not the most end-user-friendly language and the R Studio is not the best development environment, but those libraries can solve all your problems immediately. And that’s something I really like.\nFor sure you can find appropriate library among those 10 thousands libraries\nAbsolutely you are right. It is not easy to find the right library among. Fortunately, there is a huge community around R and it’s very active. So you just ask a question and will be answered pretty promptly. And besides that there are golden standards like ggplot, you just have to follow the main path, and you will be most of the time fine.\nDid you have a situation with your work with the R language that you were not able to find an appropriate library?\nLet me think. I am… Funny thing., Two years ago I had to process some medical data about cancer. So graphical data. And turned out that there is a R library specifically for this ready to be used. So no – to answer your question –no- I always found one.\nSo, it was already someone who wrote the library down.\nYes\nHow sure can you be about those libraries? Are they tested somehow? Or just people from the forum tell you that is worth trying because you cannot be sure about the results because in the ten thousand of libraries some might not work properly.\nYes, that is the hard part of the R language. Fortunately, most of the good libraries are pretty good documented. Very extensively. For example, there is a very tiny library, just four functions for example “find me a leak somewhere in this pipe”, but documentation is 500 pages long and detailed. To assess the quality of the functions you have to understand the documentation. So yes, if you want to be sure you have to spend some time reading the technical stuff.\nI remember that you prepared a special workshop about the Machine Learning or R language in the Switzerland couple years ago. Is it true?\nIn Switzerland – not, that was not me. That’s true that I had a couple of these trainings, workshops or preconfs about Machine Learning and R or Data Science. So – not in the Switzerland, but generally speaking – yes.\nSo, now let’s change the topic. What do you think about the MVP program those days. Because it evolved over the time. You are also the MVP for twenty years?\nLess. It’s just twelve…\nJust twelve!\nSo yes, I saw some changes. In my opinion the key word in this Microsoft Most Valuable Professional program is the word Microsoft.\nNot professional… But Microsoft.\nIn contrast to community. It is a Microsoft award, not community award. If you are asking about something like that – yes it is Microsoft award and if you are ok with this you will understand this program much better.\nAnd what about the MCM program? It was great for professionals but unfortunately was killed\nYes\nVery suddenly.\nYes\nAnd nothing is…\nI do remember this funny email “Please complete your training because we are closing this program soon”. Very motivating…\nBut nothing has been proposed so far. So the program has been terminated like four years ago as I think.\nI don’t remember…\nAnd nothing has been proposed to MVP’s and other people of course instead of this program and I think it is not good.\nAnd Marcin you were the very first individual in Poland who has achieved the MVP award. You are basically the older one, the older guy in Poland with MVP title…\nActually, I was the second who got this title in Data Platform (formerly SQL Server) category. It was two years after the program started in Poland.\nOk, it was many years ago. Do you remember that moment?\nYes, Yes. I am still in touch with people from the program and from Microsoft, Microsoft Poland. Last week a girl from Microsoft who I think proposed me at that time contacted me asking questions “could you do something for me, for my client” to be precise. I said “Yes”. And then probably she had to introduce me somehow to this client. She found the very first interview about how I started in this program. And I mentioned her name in this interview and she remembered. Good, old times.\nAnd now we go from the MVP program to your hobbies. What would you like to do the most? Or what you do the most? What’s your hobby?\nOk, I am trying to spend as much time as I can actively. I am swimming, playing table tennis, squash and dancing. Seriously. Four years ago I decided that I do not spend a minute longer in front of the computer than it is necessary. No gaming. No chatting. If I am not working my computer is shut down and I am doing something active.\nAlso, I can remember you spent some time on a boat.\nYes, that’s also, that is great actually. I am sailing mostly during summer, but this is great because for a week or two you are absolutely not allowed to use the computer. You have to deal with and to solve real problems and you have to cooperate with other guys on the boat. It is like a small community. Six to ten people. And you have to be sure that each one on the boat will do his job correctly. Otherwise, you will not manage. So that is something. And this is a nice hobby and I am really happy that Andrzej, one of our colleagues, got addicted to sailing due to me.\nYes, and he is sailing right now. He is not with us, he is sailing somewhere on the sea. So basically you have answered also on the next our question “what about your work-life balance”.\nYes.\nPretty easy for you. If you don’t need to work you will just close your computer and that’s it.\nI still would like to answer this question if I can.\nYeah, great. Please, go ahead.\nSo from my perspective it is changing. Ten years ago it was more work and less life. That was not good. Now I am trying to balance this in the other way. And my tipping point, the sweet spot will be probably to spend as much time not working as working.\nOk, fair enough, yes.\nBut it is still not the case. I decided at the beginning of this year to work less. And I even said to myself that I will earn less. No problem, no worries. But I failed on both…\nMarcin what kind of hints would you like to say to young people who want to start working on the market with SQL Server on data platform or that kind of stuff.\nSo first they are already in the half way to success, because they started working with data right now. And looks like that if you are a data specialist you are lucky. And to be honest, SQL language is the most popular data language and it will not change soon in favor to Python, R or U-SQL. That is my opinion. SQL is just so good in data manipulation.\nIf you are good in this and you mentioned that it is not too hard to be good in SQL as a language. That is the best start to learn more advanced – in my opinion, just more focused on particular tasks - languages or tools. Just with SQL you are already a professional in this world and my advice is to be brave enough to accept professional challenges. So if somebody asks you if you go and conduct this training in a foreign language or if you go there and deliver a speech on a conference or whatever else.\nSo your answer would be always “Yes, absolutely yes”.\nMost of the times – yes. Sometimes you have to be, you have to know your capacity, let’s say. Otherwise it would be silly. The worst thing is that if you fail because you are too eager thinking “I am the smartest guy on this planet”. Be patient and the right time will come – no worries. But if you failed miserably at the first attempt it might change your attitude.\nSo this is another kind of balance between the optimistic and realistic approaches to challenges.\nBasically, how much time do you spend learning something new and how much time do you spend making your business?\nThe best thing is when you can earn some money learning. That is the greatest occasion. It is not a joke because I deliver trainings from time to time, and most of these trainings are prepared for the specific customer needs. I have to prepare it.\nMy first point is the best part of learning is to have the discussions with people, the attendees. And you can learn a lot from them. They are asking very interesting and hard questions and this is the way I am learning a lot. The second part is learning from colleagues during conferences. This is also nice because it is two - three days like a free weekend but you are still learning – officially.\nAnd actually, you are learning. So, to answer your question Kamil there are some recommendations done by very smart people in business, that right now in IT an employee should spend at least one day per week learning, not working. At the end it will provide more profit to the company. And I think I am a little bit more profitable as I am learning – I don’t know – at least 10 hours per week, probably two days per week, something like this.\nBack to the old times. When did you start interesting in SQL and why SQL?\nBecause it seemed easy at that time. Now we are talking about really old times. I was thinking about how to find an easy job because my first university degree is not IT. So, I was thinking that maybe I can use those logic lessons from my university to learn something similar. This is one part of the story. The second part is that actually during my studies I had to earn some extra money and one of the very first projects were to develop a database in Microsoft Access for a small company so I had to learn SQL. Since then I know how good and powerful language it is.\nFrom that product, I mean Microsoft Access, it was easy to transit to SQL Server\nYes, you just have to press the button “Migrate to SQL”.\nAnd the language is almost the same. And if you know one dialect of SQL it is fairly easy to go to other engines…\nYes\n\u0026hellip; like Oracle, MySQL, Postgres \u0026hellip;\nAbsolutely you are right because the point is the logic behind. You are saying what do you want and not how to do this stuff.\nDo you have any animals at home?\nYes! I do have. Now I have two hamsters. That’s it. I am travelling a lot so larger pets are not the option I think. Gold fish would also survive. But no, we have two hamster and they are pretty nice. During the nights they are travelling across flat which is relatively large. They are going from room to room and it is always nice to hear them.\nDuring the night\nYes\nSo you have that kind of animals which can survive during your trips around Europe and the World…\nSure.. You would be surprised how much of the extra food and stuff they manage to prepare… They are pretty clever.\nRight now let’s talk about the community and your approach. If you have a speech, let’s say – international conference – and you know it is going to be tough. Because when you speak it is always tough. How do you prepare yourself?\nSo after some years of delivering speeches during meetings and conferences what help me the most is to feel confident. And there are different ways to achieve this state of mind. First, not that easy, is to prepare well – to know your topic. And actually by “know” I mean “know” – not the slides and demos but the logic behind all those stories. If you are well prepared you will be confident and people like to hear confident speakers.\nBut there are also some other techniques. Not always they are applicable but they work. So you should be familiar with the stage if possible. You should go there earlier, see the audience, spend some time in this room. This will help as well.\nBefore your speech at least some minutes before, and check everything. And after this you will still have two or maybe five minutes to relax and talk to the audience. It will help a lot. People will be happy. You will be relaxed and self-confident. So self-confident is my answer.\nWe know that you have many workshops and speeches during many conferences. But the question is if you are still learning something new during your work?\nOh yes. Absolutely. Microsoft is taking care of this. Microsoft is trying hard to find new things that are worthy to learn. Actually I am not joking. 30 minutes ago Michael Rys had this nice talk about the USQL and the latest refresh of the Data Lake Analytics. This is one of many relatively new, constantly changing, topics.\nSo I am still learning R and Python because they are evolving. Cognitive Services and AI (Artifical Intelligence) – just this – will keep you busy for years. The only problem is that I think you are not able to catch up with the current development in this area.\nLet me explain. I am studying some papers dated 2015 from universities about new algorithms and techniques and how to apply the algorithms to solve some complex tasks. And just two years after you can start playing with those algorithms using tools like TensorFlow, or Cognitive toolkit. Just two years from the university to production. So yes, I definitely still have a lot to learn.\nAre you a perfectionist?\nWell, in comparison with some others guys I know – definitely not.\nBut in the other hand and we had this discussion previously. I do like to feel self-confident and my way to achieve this is to make sure I can answer all the questions or solve the task in a proper way. This is the reason why I am learning because I just like to know. So in this respect – yes. But I am ok with people that are better than me and I know a lot of them. They are better than me in lot of things.\nSo in your opinion you are not but you are at least trying to be.\nAnd looking at the past what achievement are you satisfied the most?\nWell… it might be silly because maybe you know or maybe you not but I am still a Microsoft Certified Trainer. So the MCT. I am not delivering that many trainings per year, probably only one. For the last couple years… And this is perfectly fine because I am playing according to the program rules.\nBut 15 years ago I got this MCT title and to be an MCT you had to pass some exams before, the professional exams. But it was MCT what was the turning point. I started working differently. More interesting things to do and this slowly changed to the point where I am right now. I have probably never be employed by any company longer than two months. Maybe just after my studies. But now I am pretty sure that self-employment is good for me. It might not be good for everybody for whatever reason but it is good for me. And being a self-employed it’s good to be recognizable and those titles helped me a lot, at least at the beginning, to achieve current position.\nSo that would be also a good advice to young people. I mean start using your basic knowledge in your work then do some certificates and extend your knowledge every day.\nYes\nBut certificate does not mean knowledge in most cases. So you have to be sure you can prove your knowledge just by taking an exam and having the certificate. It should not be the opposite.\nYes, absolutely, very true. I just wanted to say that right now there is a lot of opportunities to get a good training online from the best universities, Coursera, Pluralsight, Edx, stuff like that. These courses are free or relatively cheap, and very good quality, if you spend some time and search the one you are interested in. There are some programs related to these courses. Right now is way easier to be a very good professionals than 10 or 15 years ago.\nI absolutely agree with both of you because that’s why I have just started my sentence that young people should use their basic knowledge in a practice – working.\nYou are right. Use any opportunity, like kaggle challenges, if you are playing with the AI. There are lot of them. In most of those courses you require to deliver a project. So not just pass a test but actually do something that work\nAnd if you are talking about the AI what do you think Marcin what will change the most in the next 5 years, maybe 10 years?\nIn the next 5 years probably everything. Maybe I am over optimistic but I am a beliver. The AI is the fourth technical revolution that happens right now. Most of the IT companies know that already and we are spreading this knowledge to other markets pretty quickly. So five years is a lot. I cannot answer this question. Even right now we are pretty close to build a model that can build models cleverer than the original one. It is happening right now. When we pass this border everything will change in the way we cannot foresee right now.\nAnd we are in luck to be in the middle of this change I think.\nYes, I am pretty sure. Going back your question about young people advice …\nI am not advising to young people to be truck drivers. It is risky business right now. Taxi driver as well. Or interpreter, translator. All those jobs can be reduced right now. Working in call center - and we have this issue in Poland, we are the biggest call center in the Central Europe - all those people will be pretty soon replaced by the machines, bots. So those changes will happen within the next year or two.\nMarcin, I’m holding in my hand your book. It is Data Science and Machine Learning. The book is written in Polish and I just wondering are you going to release this book in English? There are a lot of useful knowledge and I think the wider audience could use the book.\nFirst – thank you for what you have said. Second – maybe – depend also on a publisher. It is not only my decision. Third thing is that I like this book. I spend some time gathering the materials and trying to write as clearly as I could about this Machine Learning stuff, Azure Machine Learning Studio and R language but to the same work in English – I am not so sure. Maybe it will be more fun starting working on something new, but you are right – maybe this time I should start from beginning in English\nSo at the end of the conversation tell us where can we find you. Facebook, twitter…because people might like to ask you some questions.\nFacebook of course – Marcin Szeliga but it is relatively hard to find me by the foreigners. You can also try to search on the MVP website. There is an official MVP website and if you try to write down “Szeliga” or “Marcin” you will see my picture on the Data Platform MVP from Poland. You can find me on Data Community Poland. I am trying to be active. But please if you have any questions or ideas or would like to share your opinions than please contact me marcin.szeliga@datacommunity.pl – that is my email address and you can find me on Facebook and LinkedIn.\nOk, great. Thank you very much Marcin.\nThank you!\n","date":"2018-01-12T10:55:45Z","image":"/uploads/2018/01/IMG_6677_fix.jpg","permalink":"/2018/01/asf-005-marcin-szeliga-interview/","title":"ASF 005: Marcin Szeliga interview"},{"content":"In today\u0026rsquo;s article I\u0026rsquo;d would like to focus on Slowly changing dimension, aka SCD. Dimension is a word excerpted from data warehousing as such. Not without a reason SCD is used very often in terms of Data Warehouse (DW) topics and can be use for audit purposes in OLTP systems.\nThe goal of the article is to review the types of Slowly Changing Dimension as there are few version of them.\nWhy do I need SCD? The data is changing over time. You might not care about historical version of data in your OLTP database, but it might be business requirement in order to maintain all historical changes. In other words, you must be able to show, in a sales report, a correct regional office of a sales person, no matter when the person has changed the region. In that case we must keep the data before change and after the update.\nHow Transformations Work? There are few types of SCD.\nDepending of SCD type you can keep none, one or more historical changes of rows.\nThe idea is simple: every time the row is changing, a old version (before UPDATE or DELETE) of row will be kept in SCD table.\nThe implementation of these things is not always easy. Let\u0026rsquo;s have a look at examples.\nAs an example of source data I will use the following data row:\nSCD Source data example In the example above, CustomerId is a PrimaryKey in source (OLTP) database.\nWe introduce surrogate key (CustomerKey) in DW to avoid duplication of original CustomerId and to optimize performance, especially when source system uses character or complex keys.\nSCD data in Data Warehouse example SCD Type 0 This type doesn\u0026rsquo;t keep any changes of attributes. It keeps only the original value without any changes. It used mainly for attributes which can\u0026rsquo;t be changed, like BirthDate.\nSCD Type 1: overwriting the history for an attribute This method can not track the changes in data, and overwrites the old data with new. Once our customer change his address, new address would overwrite:\nSCD Type 1 As you can see - it\u0026rsquo;s very simple to implement and maintain the process - it\u0026rsquo;s simple update. In other hands, the drawback of this approach is lack of row\u0026rsquo;s history in Data Warehouse.\nSCD Type 2: adding new rows The methodology is preserving the history by adding new rows and there is no limit on number of keeping changes.\nSCD Type 2 Additionally, you can decide in which way you want to know older rows by maintain current flag column and/or UpdateDate:\nSCD Type 2 with current flag or version of row which will be incremented sequentially:\nSCD Type 2 with version SCD Type 3: additional historical columns In this approach you can manage a limited amount of history by adding additional columns for tracking attributes.\nWanting to have previous and current city of customer you can implement columns: PreviousCity and CurrentCity. There will be only one row in Data Warehouse (DW) per source database therefore Type 3 has limited usability and is less popular than Type 1 and 2.\nSCD Type 3 You must be careful if you want to use Type 3 of SCD. Make sure that Business Analyst and/or your Customer are aware of history\u0026rsquo;s limitations.\nSCD Type 4: Separate historical table This type uses separate tables for keeping the current records and historical changes. It worth to use when:\n• Dimension is changing pretty often or\n• Dimension is multi-million rows table\nSCD Type 4 current table SCD Type 4 - historical table Margy Ross from Kimball Group calls such tables as \u0026ldquo;mini-dimension\u0026rdquo;.\nSCD Type 6: This approach is numbered as 6 as 1+2+3=6 as well 1x2x3=6. Therefore it\u0026rsquo;s a combination of these three types:\n• Keep both previous and current values for attribute (as Type 3, look at columns: \u0026ldquo;HistoricCity\u0026rdquo;, \u0026ldquo;CurrentCity\u0026rdquo;)\n• Overrides all records for current version of attribute (as Type 1, look at column \u0026ldquo;CurrentCity\u0026rdquo;)\n• Keeps historical version of attribute (as Type 2, look as columns: \u0026ldquo;IsCurrent\u0026rdquo;, \u0026ldquo;StartDate\u0026rdquo;, \u0026ldquo;EndDate\u0026rdquo;)\nLet\u0026rsquo;s have a look how the row is being changed and how its affect on DW. The first loading of the row:\nSCD Type 6 (step 1) Rows in DW\u0026rsquo;s Customer table following first customer relocation:\nSCD Type 6 (step 2) Rows in DW\u0026rsquo;s Customer table following second customer relocation:\nSCD Type 6 (step 3) Which Type should I choice? As a professional consultant I reply professionally:\nIt does depend Ask your business team, business analytics or your customer: What they need in data warehouse.\nMaybe I didn\u0026rsquo;t say it loud enough, but in the same table you can apply different SCD Types to each column.\nFor instance: Type 2 to the City column, Type 1 to the BirthDate column. All in the same table.\nTherefore ask them for every dimension:\n• which attributes are constant,\n• which attributes are variables?\n• which attributes should be fully-tracked, which should keep only previous and current version?\n• How many rows are in table right now / are expected in the next 10 years?\nKeep these answers as a evidence. People, especially customers, love to change their mind. You know what I mean.\nIf you still not sure which one to be chosen, remember that:\nKeeping a history of the changes are good for you Simply because the requirements could change.\nBeside of types 0 and 1, in my opinion, the most popular are Type 2 and Type 4.\nHow to load data into SCD? Good question. But this topic is a completely different story.\nBefore I prepare next post about it - have a look on this tool:\nSCD Merge Wizard\nThe tool helps you to create, manage and maintenance of mapping between source database (e.g. staging) and target database (Data Warehouse). Furthermore, with this approach you will use MERGE statement to perform loading operations using configured SCD Type for each column.\nIn further posts, I will explain how to apply different SCD Types in this tool.\nGood luck with your loadings,\nKamil\n","date":"2018-01-09T14:36:44Z","image":"/uploads/2018/01/change-2850274_1280.jpg","permalink":"/2018/01/slowly-changing-dimension-scd/","title":"Slowly Changing Dimensions (SCD)"},{"content":"Thanks for visiting the page. Hope you will find some time to read great articles I have found for you.\nHere you are - this week readings. Start reading on Monday morning -\nZones in Data Lake\nYou know I do really like ADL concept + USQL hence you should not be surprised that a new great article is mentioned here in the first place. Take a look there and you will see how the Data Lake is organized internally. Plus you should read other articles from @SQLChick blog about the ADL.\nDatabase scoped optimizing for ad hoc workloads\nJoseph Sack is back explaining how the \u0026ldquo;optimize for ad hoc workloads\u0026rdquo; works at the SQL Server instance scope. What is even more important the OPTIMIZE_FOR_AD_HOC_WORKLOADS option now enables the database scope in Azure SQL Database.\nMaking parallelism waits actionable\nAnother great entry by Pedro Lopes. He mentioned this during the PASS last year (November). The goal is how to make parallelism waits more actionable. You might guess that this is about the lovely CXPACKET wait type. Take a while and read about an actionable CXPACKET and negligible CXCONSUMER.\nBreakpoint debugging of Spark jobs on Azure HDInsight\nTired of reading so far? I can understand that. It is Monday morning\u0026hellip; Take a break and spend 11 minutes 13 seconds watching the video about how you can use your favorite developer tools to debug Spark applications.\nSQL: Best Way to Scale SQL Server Database Performance\nGet Greg Low\u0026rsquo;s advice about SQL Server database performance. Greg has done an excellent job - he has rated performance problems by cause and has explained what is the big problem. I agree with him.\nPartition and scale in Azure Cosmos DB\nYou can read and watch this blog post. You can read because the partitioning and scaling topic is explained in great details. Plus bonus video where Scott Hanselman talks to Azure Cosmos DB Principal Engineering Manager, Shireesh Thota about partitioning and partition keys\nCheers,\nDamian\n","date":"2018-01-07T22:11:48Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2018/01/last-week-reading-2018-01-07/","title":"Last week reading (2018-01-07)"},{"content":"History of certain photos\u0026hellip; Oughh\u0026hellip; some time has gone, I know. Anyway, only now, at the beginning of new year and exactly a half year after the event, I found a piece of time to sort out all my photos of the year 2017. After that work I realized that photo album from the event are already on my Facebook profile (as all albums on this blog comes from Facebook and they are synchronizing automatically), hence I could put this post before, without unnecessary delay. OK, nothing lost - I needed work this out anyway.\nWhat happened in 2017 in Lingen? The event has taken place on Friday, 2nd of June 2017 and it was my first conference in Germany (AFAIR). The time right now is a very good moment for reminiscences. Why? Because the Call for Speakers is still opened until 31st of January 2018, so not many time left. Especially as the organizers have opened entirely new track called \u0026ldquo;NEWCOMER\u0026rdquo;. If you are a new speaker or need help - submit your session on this track. It\u0026rsquo;s including a speaker mentor and moral support.\nLast year (2017) during the event was/were/or I experienced:\nThird edition of SQLGrillen (the very first time has had a place in 2015) 4 tracks, 28 sessions (over 130 session submissions) and 140 attendees Stephanie Locke [T] birthday Alex Yates [T] has received breathtaking news (MVP award) I saw dbatools and Pester (for the first time) in action (actually, I started loving #PowerShell) BBQ dinner after the event was absolutely awesome! as always, I met some old and new friends - best regards to: Chrissy LeMaire [T], Cláudio Silva [T], Jens Muether [T] a trip from London to Germany by car is probably not the best option (thanks Janusz [T] for company) Thanks William [T] for creation such as great event!\nMore info: SQLGrillen 2017, @SQLGrillen, website\nPhoto album [srizonfbalbum id=7]\n","date":"2018-01-04T18:17:41Z","image":"/uploads/2018/01/SQLGrillen.png","permalink":"/2018/01/sqlgrillen-2017/","title":"SQLGrillen 2017"},{"content":"Yeap, this is last day of the year 2017. So, I do not expect too many readers of this post - almost everyone is preparing to go to New Year Eve Party, meeting with friends or spending their time with family at home. Probably not reading blogs, right? Therefore this post gonna be quite short.\nCopy data from Azure blob storage to Azure cosmos DB\nQuick tutorial how to copy data to Cosmos DB\nWhat has happened in the DAX world in 2017\nMarco Russo summarises changes (past and future) from DAX perspective\n31 Days of #CosmosDBTrivia in December\nIt is not a single post but series of posts on Twitter about #CosmosDB\nTop 10 Retail Banking Trends and Predictions for 2018\nGood reading at beginning of the new year, not only for those who are working in financial market.\nChecklist for Finalizing a Data Model in Power BI Desktop\nDouble check your data model in Power BI before you pass the reports to business\n24 Data Science, R, Python, Excel, and Machine Learning Cheat Sheets\nGood starting point if you interested one of these languages/trends\nSQL Server Internals: In-Memory OLTP\nRedgate \u0026amp; Kalen Delaney offer the second edition of her book about In-Memory\nWhat’s So Bad About Shrinking Databases with DBCC SHRINKDATABASE?\nTalking about why not shrinking databases is never enough. Read Brent Ozar\u0026rsquo;s post if you\u0026rsquo;re not sure the real impact.\nTomorrow, the new year will commence. I wish you all - implementation of all your goals and dreams, many successes and obviously\u0026hellip; more data to work with, health, smile and happiness also in you work place!\n*** Happy New Year 2018 !!! *** Kamil\n","date":"2017-12-31T14:38:53Z","image":"/uploads/2017/12/fireworks-justin-sinclair-443556.jpg","permalink":"/2017/12/last-week-reading-2017-12-31-happy-new-year/","title":"Last week reading (2017-12-31) \u0026 Happy New Year!"},{"content":"Sometimes a message coming from SQL Server engine can be misleading and confusing.\nI had such cases many times and decided to put a post every time I meet it again.\nFailed to initialize sqlcmd library with error number -2147467259 When I was working on one SQL Job - its constantly reporting an error:\nMsg 22050, Level 16, State 1, Line 58\rFailed to initialize sqlcmd library with error number -2147467259. Although SQL Job has had only one step and called one specific stored procedure, but that stored procedure contained many nested procedures.\nTherefore, at the beginning it was hard to find a culprit of the error as each step was dependent of the previous one.\nAnyway, exclusion one by one brought an effect. I found a nasty step. Bastard Stored Procedure.\nCulprit It doesn\u0026rsquo;t contains too many business logic, but send alert by email in specific condition.\nThe send block looked like below:\n1 2 3 4 5 6 7 8 9 10 11 12 EXEC msdb.dbo.sp_send_dbmail @recipients = \u0026#39;recipient@azureplayer.net\u0026#39; ,@subject = @Title ,@body_format = \u0026#39;HTML\u0026#39; ,@body = \u0026#39;You can find the details in the attachment.\u0026#39; ,@attach_query_result_as_file = 1 ,@query = \u0026#39; SELECT TOP 10 [BusinessEntityID], [PersonType], [Title], [FirstName], [LastName] FROM [Person].[Person] WHERE [EmailPromotion] = 2; \u0026#39;; I have confirmed that the statement was a reason of the error.\nSpending some time on the research I found eventually that the root cause could be a result generated from a query.\nIndeed. Giving up with the query - everything works:\n1 2 3 4 EXEC msdb.dbo.sp_send_dbmail @profile_name = NULL, @recipients = \u0026#39;recipient@azureplayer.net\u0026#39; @subject = @Title, @body = \u0026#39;Some email body here...\u0026#39;; Hence, something is bad with the query even though the query is executed successfully.\nSolution #1 Then I realize that the query uses 2-part object names, so I have added database name to query giving fully qualifying table name:\n1 2 3 4 5 6 7 8 9 10 11 12 EXEC msdb.dbo.sp_send_dbmail @recipients = \u0026#39;recipient@azureplayer.net\u0026#39; ,@subject = @Title ,@body_format = \u0026#39;HTML\u0026#39; ,@body = \u0026#39;You can find the details in the attachment.\u0026#39; ,@attach_query_result_as_file = 1 ,@query = \u0026#39; SELECT TOP 10 [BusinessEntityID], [PersonType], [Title], [FirstName], [LastName] FROM [AdventureWorks2012].[Person].[Person] WHERE [EmailPromotion] = 2; \u0026#39;; That\u0026rsquo;s work!\nSolution #2 Another solution of that problem is to provide the optional @execute_query_database parameter:\n1 2 3 4 5 6 7 8 9 10 11 12 13 EXEC msdb.dbo.sp_send_dbmail @execute_query_database = \u0026#39;AdventureWorks2012\u0026#39; ,@recipients = \u0026#39;recipient@azureplayer.net\u0026#39; ,@subject = @Title ,@body_format = \u0026#39;HTML\u0026#39; ,@body = \u0026#39;You can find the details in the attachment.\u0026#39; ,@attach_query_result_as_file = 1 ,@query = \u0026#39; SELECT TOP 10 [BusinessEntityID], [PersonType], [Title], [FirstName], [LastName] FROM [Person].[Person] WHERE [EmailPromotion] = 2; \u0026#39;; Conclusion This is because sp_send_dbmail doesn’t seem to have any database context.\nNow, it\u0026rsquo;s seem to be easy. But the error message wasn\u0026rsquo;t been helping solve this \u0026ldquo;mystery\u0026rdquo; quickly.\nEnjoy! I hope that post helped you.\n","date":"2017-12-28T23:26:42Z","image":"/uploads/2017/12/businessman-2956974_1920_s.jpg","permalink":"/2017/12/sp_send_dbmail-fails-with-query-result-attached-as-file/","title":"sp_send_dbmail fails with query result attached as file"},{"content":"Thanks for visiting the page. This is a special week - the Christmas Week! Hope you will find some time to read great articles I have found for you.\nHere you are - this week readings.\nInstalling Microsoft SQL Server on Ubuntu 16.04\nI have written a post about this process last year but this was in Polish. You can still find it on Technet. You can find the English version of it on the Bobby Allen\u0026rsquo;s blog.\nNot for reading. This is a presentation done by Bob Ward. It takes only 5 minutes to watch so this might be actually a good option if you are tired during the Christmas morning\nHow to use Tableau with SQL Server on R and Python\nIf you are curious how to work with Tableau and make use of data from SQL Server then go and read this.\nChecklist for Finalizing a Data Model in Power BI Desktop\nMellisa Coates (aka @SQLChick) explains why data modelling in important in PowerBi. This is actually a checklist rather than an article but you should use it any case you work with PowerBi.\nMicrosoft SQL Server R Services\nHere you can find the entire series dedicated only to R Services. This is done by Niels Berglund whom I have met in 2008! And he has been blogging since then!\n20 Best SQL Server Monitoring Tools for all SQL Servers\nThis is a little bit old article about monitoring tools you can find for SQL Server.\nZIP SOURCEFILES IN A SSIS PACKAGE (SSIS SERIES)\nAn article describing how to deal with compressed files using the SSIS 2014.\nDatabase Corruption - Real World Issue\nThe real issue that happened and lesson learned. Always worth to read and learn\nCheers,\nDamian\n","date":"2017-12-24T21:50:30Z","image":"/uploads/2017/12/P1533428.jpg","permalink":"/2017/12/last-week-reading-2017-12-24-aka-merry-christmas/","title":"Last week reading (2017-12-24) aka. Merry Christmas!"},{"content":"How Can I Become A Microsoft MVP?\nMany people ask how to achieve the title, so Kevin Kline [T] answers. Forget, it\u0026rsquo;s not a recipe - just do what you love doing.\nTry mssql-cli, a new interactive command line tool for SQL Server\nSQL Server Team announced the Public Preview release of mssql-cli, a new and interactive command line query tool for SQL Server.\nSQL Vulnerability Assessment now available for SQL Server 2012 and up\nHas just been released on SSMS 17.4, check it out.\nADL Tools for Visual Studio Code (VSCode) supports Python \u0026amp; R Programming\n#VSCode supports your #USQL scripts with #R and #Python\nConnect(); 2017: SmartHotel360 Demo Apps and Architecture\nSmartHotel360 is a fictitious smart hospitality company - Architecture diagram and sample code\nLatest \u0026ldquo;SQL in the City\u0026rdquo; virtual event straight from Cambridge\n15 Examples of Internet of Things Technology in Use Today\nExamples of the technology in action nowadays.\nHP laptops found to have hidden keylogger\nWatch out! Your key-pressings might be logged \u0026ldquo;by mistake\u0026rdquo;\nNew in SSMS – Always On Availability Group Latency Reports\nFind the bottleneck in the log transport flow between the Primary and the Secondary replicas\nKnee-Jerk Wait Statistics : SOS_SCHEDULER_YIELD\nLet\u0026rsquo;s have a look at that wait and refresh for what it\u0026rsquo;s responsible\nBuilding a modern campus\nMicrosoft is building a sustainable and modern campus in Silicon Valley (video)\n","date":"2017-12-17T19:00:00Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2017/12/last-week-reading-2017-12-17/","title":"Last week reading (2017-12-17)"},{"content":"T-SQL Tuesday #97 is hosted by Mala Mahadevan [B|T] and this time the topic is \u0026ldquo;Setting learning goals for 2018\u0026rdquo;. Mala ask about our next year\u0026rsquo;s plans in terms of learning: what we want to learn, how and when, and our plan to improve on what we\u0026rsquo;d learned.\nIt\u0026rsquo;s not easy to point out what I\u0026rsquo;d like to learn in next year. Not due to lack of ideas. On the contrary: TOO many ideas! Also, the time before Xmas is a hard time to making a plan for the next year.\nAnyways, let\u0026rsquo;s try to predict.\nWhat do you want to learn? That might surprise some people but I will still be learning and improve my English skills. I really see my progress after over 2 years of living in London now. Although I\u0026rsquo;m able to talk to people (almost fluently) and make a presentation in English, still feel that there are many things to do, many words to pick up, many phrases which could be used.\nThe second thing I would like to improve is my presentation skills. Since I\u0026rsquo;m doing several presentati/s are being got very well by attendees - I\u0026rsquo;m still could be better and amend certain things. But it\u0026rsquo;s a continuous process, right?\nFrom the technical perspective. Yeah\u0026hellip; many things.\nIt\u0026rsquo;s additionally harder because I learn and read many articles every week (check Last Week reading series).\nI would like to improve my hands-on experience (\u0026ldquo;from the field\u0026rdquo;) or learn more about such things like Tabular Model, R language (thanks Steph, Cosmos DB, Azure Data Factory (including v2 and Integration Runtime), Azure DataWarehouse, deeper dive into Columnstore Indexes. Very likely I\u0026rsquo;ve forgotten something but I will review my list at the beginning of 2018 as I\u0026rsquo;m going to get off almost entire January to rest, learn, plan the year and do my private stuff.\nAnd obviously - I will choose what I will focus on. All mentioned things are impossible.\nHow and when do you want to learn? There are plenty of sources nowadays. You know that. The Internet comprises an unlimited set of knowledge. I have already my favourites websites and YouTube channels. Furthermore, I\u0026rsquo;ve taken part in many conferences across Europe and even another continent (this year), thus I can use those opportunities to learn from my friends and other speakers. / Moreover, I really like to read books. The paper once. Honestly. In terms of that, I\u0026rsquo;m the old-fashion guy. This year I began reading book\u0026rsquo;s friend of mine - Marcin Szeliga [T]. He is a guru of Data Science and Machine Learning, and for me will be easier to finish the book as it\u0026rsquo;s in my native language, Polish.\nTherefore, there is no lack of capabilities. Only lack of time to use all of them.\nHow do you plan to improve on what you learned? My blog has started 5 months ago. So, it is a baby. I want to feed it. Feeding it regularly. I have managed it so far and I will try to do this in the next year introducing new ideas. The baby must be growing. That gonna give me wild satisfaction and let me gain my first aim - learning English. 2 in 1. Brilliant, isn\u0026rsquo;t it?\nDuring the passing year, I\u0026rsquo;ve been 12 times at varies conferences (look here) - both as an attendee and as a speaker. Next year I\u0026rsquo;m going to continue that activity, but with a bit less frequency. Although I love meeting all my friends from #SQLFamily and be on many conferences - I must be more precisely next year and select only part of them. It gonna be difficult.\n","date":"2017-12-12T22:08:59Z","image":"/uploads/2017/12/books-2412490_650.jpg","permalink":"/2017/12/t-sql-tuesday-97-my-learning-plans-for-2018/","title":"T-SQL Tuesday #97 – My learning plans for 2018"},{"content":"Thanks for visiting the page. Kamil started the thread last week. We think this could be a very good thread not only for you but also for us. We are looking for interesting topics all the time and would like to share all the findings with you\nWhy is it important for us, too? Last week Kamil said:\nSometimes I need to back to an article which I\u0026rsquo;ve been reading some time ago. But\u0026hellip; I don\u0026rsquo;t remember where it was. This series certainly help me. Although I hope it help you as well. Have you read anything interesting too? Share it. Share it here (in comments), send me an email, share it on your Facebook and/or Twitter. Anyhow. Simply - share it.\nAnd he is right. We need to have a repository of the stuff.\nHere you are - this week readings.\nDebugging Azure Data Lake Job Failures Made Easy (part 1) – Debug U-SQL job failure of C# custom code\nI have felt in love with ADL and USQL long time ago. And this is a \u0026ldquo;must read\u0026rdquo; article if you need to do debugging of the code.\nThe future is quantum: Microsoft releases free preview of Quantum Development Kit\nMaybe you do not know but quantum physics is my passion. Now I will be able to have a toolkit and play it! You can do it, too!\nPower BI Desktop December Feature Summary\nTake a while to read the latest news from the Power BI world. This time the December Update\nDevOps for Data Science – Continuous Integration\nBuck Woody talks about DevOps and Continous Integration. How does he approach the problem of DevOps in the Data Science?\nRunning SQL Server in the Cloud: 2017 Edition\nBrent Ozar explains the options of using the SQL Server in the cloud\nGetting Started With Microsoft Azure Functions Using Visual Studio 2017\nSomething easy at the end. Learn how to start developing Azure Functions. I love them. I use them. You can do it, too!\nCheers,\nDamian\n","date":"2017-12-12T01:06:34Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2017/12/last-week-reading-2017-12-10/","title":"Last week reading (2017-12-10)"},{"content":"\nLink to this (past) event: http://www.sqlsaturday.com/687/\nPictures [srizonfbalbum id=12]\n","date":"2017-12-10T10:10:52Z","image":"/uploads/2017/12/IMG_8281.jpg","permalink":"/2017/12/sql-saturday-687-slovenia/","title":"SQL Saturday #687 Slovenia"},{"content":"Induction This is a new series. Posts in that series as summary of last week reading.\nSo long I was thinking about it whether is it good idea to post something like that or not? And you know what? Almost every time when I\u0026rsquo;m not sure something - I\u0026rsquo;m going to do that. To try. To taste. To experience.\nSometimes I need to back to an article which I\u0026rsquo;ve been reading some time ago. But\u0026hellip; I don\u0026rsquo;t remember where it was. This series certainly help me. Although I hope it help you as well.\nTo find out more, read something new, extend knowledge.\nHave you read anything interesting too? Share it. Share it here (in comments), send me an email, share it on your Facebook and/or Twitter. Anyhow. Simply - share it.\nSQL Server 2017: A proven leader in database performance\nNew Hewlett Packard Enterprise (HPE) benchmark results continuing to show SQL Server’s leadership in many areas.\nThe Wait Is Over – SSIS 2017 Is Here!\nHow to set up Visual Studio 2015 with SSIS 2017 from SSDT.\nWhat is Microsoft SQL Operations Studio (preview)?\nEric Kang [T], Senior PM at Microsoft, presents new, free, multi-platform tool for SQL Server.\nBackground removal with deep learning\nDoes the Machine Learning clever enough to remove the background from a picture? Find out it.\nCommon Azure Cosmos DB use cases\nOverview of several common use cases for Azure Cosmos DB. What are the benefits from each of them?\nMaking parallelism waits actionable\nPedro Lopes [T], during PASS Conference, announced that CXPACKET will be split out introducing new wait: CXCONSUMER\nAnnouncing the Azure Database Migration Service Public Preview!\nThe Database Migration Team [B] have migrated 30K databases in the past six months using DMS service and tools such as Data Migration Assistant (DMA), the Database Experimentation Assistant (DEA) and SQL Server Migration Assistant (SSMA).\nPower BI Security whitepaper\nAn updated whitepaper (November 2017) includes coverage of Power BI Premium.\n","date":"2017-12-03T23:47:38Z","image":"/uploads/2017/12/newspapers-390.jpg","permalink":"/2017/12/last-week-reading-2017-12-03/","title":"Last week reading (2017-12-03)"},{"content":"When you doing something first time, predominantly jitters happens, right? Fine, it was not my first conference, I wasn\u0026rsquo;t in a speaker role this time, etc. However, certain voice in my head were asking me \u0026ldquo;what do you going there for?\u0026rdquo;. That was that bad guy.\nThe other one, the good one, was repeating over and over:\n\u0026ldquo;come on! you\u0026rsquo;ve never been on such big conference. Even better, you\u0026rsquo;ve never been in the USA. Never exceeded the big water.\u0026rdquo;\nIt\u0026rsquo;s an excellent opportunity to be there, can see everything, meet some people you only contacted via tweets by now, meet Microsoft\u0026rsquo;s folks and, last but not least, make new acquaintanceships.\nTravel Travel was not bad at all. Furthermore, I must admit that it was very pleasant. I had a direct connection from London to Seattle (some folks from the US still don\u0026rsquo;t believe me in that), quite fresh one. It was Norwegian\u0026rsquo;s flight on board of Boeing 787. Yes, that one - aka Dreamliner. To my amazement, I had free all seats in my row. Believe me, I have paid only for one!\nBeing in Seattle I had couple days and I spent them to sightseeing. This post is not about it, so I only mention that Andre Melancia [T] and myself had good trip to Boeing Factory and land/water tour #onaduck (don\u0026rsquo;t laugh!).\nFirst-Timer True. First timer. That was my first title on my badge. I\u0026rsquo;m still feeling very good doing something for the first time. Like this experience. Being co-organiser of the biggest SQL conference in Poland, I was feeling a little anonymously at PASS Summit, the biggest SQL conference in the world.\nThe event has a special meeting for such first-timers like me: \u0026ldquo;PASS Summit 2017 Buddy Program\u0026rdquo;. The program aims to help First-Timers get the most out of first conference experience. Firstly, a title buddy, met us on-site and provided the information we needed.\nAs a part of it, there was also the First-Timer Orientation \u0026amp; Speed Networking event hosted by Cathrine Wilhelmsen [T|B] with Bill Graziano (T) this year.\nWe have had a lot of fun during introductions ourselves or attempts answering the questions displayed on the screens to an individual who was seating in front of you. We had only 3 minutes per a round, and in that way, we got to know 10 new people in 30 minutes.\nKeyNote Day 1 Although I was late, I went to the first row as I always like to see well the actors on the stage. I\u0026rsquo;m little shy only.\nThe first day of Keynote was leading by Rohan Kumar who was inviting many guests on the stage. In a brief, what we heard about new features in SQL Server 2017:\nSupport for graph data and queries Advanced Machine Learning with R \u0026amp; Python Native T-SQL scoring Adaptive Query Processing and Automatic Plan Correction It worth to mention about several new features/services (all in preview phrase yet) in Azure SQL Database:\nData sync v2 Managed Instance Database Migration Service New version of Azure Data Factory The keynote contained many practical demonstrations, giving us a chance to watch the features in action. We could have seen:\nBob Ward [T] and Conor Cunningham [T] whose demonstrated how super fast performance has been achieved in SQL Server 2017 and why performance tuning will become automated job Tobias Ternstrom [T] and Mihaela Blendea [T] have showed how quickly set up, in a new way, a development environment with supported containers. \u0026ldquo;Hospital Intake Prediction\u0026rdquo; application demo during which Danielle Dean [T] has presented how scalability and high performance in Azure SQL Database can combine with built-in AI to create new and powerful insights from IOT data. Scott Currie [T], CEO of Varigence, showed off how Azure Data Factory can collaborate with BIML language to automate data movement in the cloud. Julie Strauss demonstrated new compute-optimized tier in Azure SQL Data Warehouse. Christian Wade showed how you can spread out your workload and Scale Out up to 7 read-only replicas, each with 10 billion rows, allows you to support thousand of concurrent users. At the end, Riccardo Muti showcased hybrid BI solution with Power BI Premium using interactive reports and stunning visualizations \u0026ldquo;Quite accidentally\u0026rdquo;, Microsoft has announced new tool: Microsoft SQL Operations Studio (T|W) - new, multi-platform, open-source application which empowers DevOps practice to work with SQL Server, Azure SQLDB \u0026amp; DW.\nKeyNote Day 2 First 4 minutes have belongs to Grant Fritchey who was talking about PASS itself - its growth, finances, Virtual Group Members and goals for 2018. Grant was encouraging everyone not only to learn, but also make new connection, but absolutely to use event opportunity to have fun! I agree with him absolutely.\nThen Denise McInerney briefly described her journey with PASS which continues since 2012 and her role as a VP of Marketing.\nFor the next over 80 minutes on the stage was fantastic Dr Rimma Nehme. She is an Architect of Azure Cosmos DB at Microsoft.\nDr Rimma Nehme [T] reminded me that 90% of the world\u0026rsquo;s data was created in the last 2 years and how the current trends in data looks like: Data is growing, 80% incoming data nowadays originally are in unstructured form. Florence was the original project name when CosmosDB was born about 2010.\nSo what the Azure Cosmos DB is? It\u0026rsquo;s a globally distributed, massively scalable and multi-model database service which having:\nTurnkey global distribution Elastic scale out of storage \u0026amp; throughput Guaranteed low latency at the 99% percentile Five well-defined consistency models Comprehensive SLA\u0026rsquo;s Dr Nehme went through and described the Resource Model and System Design as well as from physical perspective and from partitioning.\nOn of the cool things among CosmosDB is that it was designed to elastically scaling throughput from 10 to 100s of millions of transactions/sec across multiple regions. Another cool stuff is automatically indexed SSD storage - they just don\u0026rsquo;t ask you if you want it or not. It\u0026rsquo;s by default, out of the box. During her session, we could have understood how Cosmos DB is beating the speed of light ;)\nYou can download slides from her presentation.\nNetworking As I mentioned, each conference is not only possibilities to lea/especially great opportunity to networking, which I love to do. You can meet new people during an event or at the (one of many) evening party. During this event, my first PASS Summit, I\u0026rsquo;ve met in person the people who I knew only from blog, twitter and so on: Kendra Little [T|B], Pinal Dave [T|B]. Furthermore, I had occasion to see in live to guys from Microsoft\u0026rsquo;s crew I\u0026rsquo;ve never met before like Dr Rimma Nehme [T], Adam Saxton [T|B], Patrick LeBlanc [T|B], and the others.\nMicrosoft Clinic Another dodgy thing during the event was a Microsoft Clinic stall. The place where you could come over, find a bunch of Microsoft specialists and ask them any difficult questions from any field about an issue you have been struggling. Even if they couldn\u0026rsquo;t answer your questions on site (predominantly they do), you will get a contact to right person in Microsoft or they reply to you as soon as possible.\nI couldn\u0026rsquo;t not use this opportunity and I\u0026rsquo;ve booked myself in a queue to discuss about existing problems with SSDT whereas your solution is (huge) having a lot of circular references. So, I\u0026rsquo;m not only had a chat with guys who helped me (kudos to Dimitri Furman and Prabhaker Potharaju) to find the right guys in Microsoft, but also I could see how the Microsoft Clinic works inside. Finally, I found (Kevin Cunnane [T] and Eric Kang [T]), seen their session and we are carry on discussion by the emails.\nHopefully (I wish), certain very painful issues about SSDT gonna be resolved.\nInterviews If you\u0026rsquo;re reading this post on my blog, you probably know, there is a podcast series titled \u0026ldquo;Ask SQL Family\u0026rdquo;. I would not be myself to if I not use an opportunity to talk with some folks from the US. Damian was connected via Skype and it was more challenge for us this time. Guess why? Yes, due to the time difference: -9 hours between Seattle, WA and Poland (there is only one time zone). Anyway, new 5 episodes have been recorded with: Bob Ward [T], Cathrine Wilhelmsen [T|B], Michael Rys [T], Pedro Lopes [T], Grant Fritchey [T]. Many thanks to all you guys for your time. We were talking about SQL Server, USQL, Azure Data Factory, tools, hobbies, women in technology, work-life balance, community and volunteers. If you interested to hear on of our guests - please stay tuned.\nSummary Amazing, I\u0026rsquo;ve written much more than I\u0026rsquo;ve expected. My thoughts after my first time at PASS Summit? Impressive. It\u0026rsquo;s really big show and everything (at least from attendee point of view) was worked out in details.\nWill I go next year?\nI\u0026rsquo;m not sure yet. I would like to, because it\u0026rsquo;s really great event with dozens fantastic speakers who covering many variaties topics.\nIn the other hand in the UK, we have also very cool \u0026amp; big event (SQLBits) which is more affordable for me. Even though I had free entry tickets for the conference (thanks again Alex Whittles \u0026amp; Chris Testa-O\u0026rsquo;Neill) - all the other costs (flight, accomodation) had to be covered by myself.\nEither way, it would be hard to say \u0026ldquo;no\u0026rdquo; as I promised to Alexander Arvidsson (T) that I\u0026rsquo;ll be there next year.\nThat\u0026rsquo;s all PS. I had a chat with a few more people and sorry that I didn\u0026rsquo;t mention about all of you.\nAny comments about PASS Summit or topics described above? Don\u0026rsquo;t hesitate and leave your opinion below.\nThanks for reading,\nKamil\nLinks:\nPASS Summit 2017\nMy photo relation from PASS Summit 2017\n","date":"2017-11-20T00:12:12Z","image":"/uploads/2017/11/IMG_7538.jpg","permalink":"/2017/11/my-recap-of-pass-summit-2017/","title":"My recap of PASS Summit 2017"},{"content":"It\u0026rsquo;s photo relation only at the moment. I will write down brief relation soon. Stayed tuned.\n[srizonfbalbum id=5]\n","date":"2017-11-11T00:20:14Z","image":"/uploads/2017/11/IMG_7722_50prc_PASS_Summit_2017_Halloween.jpg","permalink":"/2017/11/photo-relation-from-pass-summit-2017/","title":"Photo relation from PASS Summit 2017"},{"content":" Introduction Mark Broadbent. Very active member of #SQLFamily, loves communicate via social media. Organizer of the very first SQL Saturday in United Kingdom – in Cambridge. Holder of very prestigious title MCM and MVP for couple years (but everybody knows that he deserved for that many years ago).\nA perfectionist who always firmly involved in everything he does.\nThis talk was recorded during SQL Saturday #632 in Cambridge, UK, on Saturday 9th September 2017.\nDo you know what joke is circulating around Mark\u0026rsquo;s MVP award and when was the first SQL Saturday in the United Kingdom? What type of mantra does Mark profess and why he thinks himself as few emotional guy? What are the Ten Bulls and what is its significance to anything?\nFind out all above and more directly from the podcast of \u0026ldquo;Ask SQL Family\u0026rdquo;.\nTranscript Kamil Nowinski: Hi, Mark.\nMark Broadbent: Hello there.\nKamil Nowinski: This is an interview that is meant to show you not only as a professional but as a normal person, show your habits, show what you do in your everyday life, maybe not very private life but\u0026hellip;\nMark Broadbent: That may be not very interesting but I’ll do my best!\nKN: You never know! But yeah, we’ll be talking about a lot of different things. So let’s start at the beginning and from a really simple thing. What is your name and where are you living?\nMB: Ok, so my name is Mark Broadbent. I’m currently living in one of the smallest, not the smallest but one of the smallest cities in England -which is called Ely. The reason it’s called Ely is historically from Middle Ages when most of the area was all under water and it was well known for its eel population and even now, it’s still well known for having quite a lot of eels. Ely is about 15 miles outside of Cambridge.\nKN: That’s why you chose Cambridge for the event.\nMB: Exactly. If I ever wanted to downscale, I’d move the event to Ely.\nKN: OK, and what do you do for the living? I know something about it but\u0026hellip;\nMB: So, I work primarily in SQL Server, as a consultant/contractor. For the last couple of pieces of work that I’ve done, they were all performance tuning-related. I’m currently working as a BI developer, which for me is kind of interesting because I’ve always tended to side on the DBA administrative side of things. I think it’s obvious to everyone that BI and business analytics are heavily on the rise (as well as machine learning), sSo it’s certainly an area I’m wanting to move into, particularly so that I can keep paying my bills.\nKN: And that’s the future of IT and everyone probably needs to go in that direction.\nMB: Yeah, well I’m still waiting for the day that a robot can do my job and then I can go and sunbathe all day.\nKN: Yeah. Let’s talk about your blog and your Twitter, Facebook activity. Are you a fan of social media?\nMB: I think if you’d asked me that a couple of years ago, I would’ve definitely say “yes”. I love the way that social media can connect us all. For instance, we knew each other long before we’ve met face to face, purely because of social media. And I think when you can put faces to people, you’re not having to waste time on that uncomfortable stage of getting to know someone. You already know what they’re about by that stage. The reason that I’ve become a little bit negative with social media is because of a lot of the political things that have been going on recently and the way that a lot of people are being shut down just because they hold completely different view. I’ve been guilty myself of posting political tweets or political posts, stating what my opinion is on things and as a result had a lot of negative comments from social justice warriors who just want to shut others down -which makes me angry. So I’ve decided to try and keep social media just purely technical-focused talking about all those irrelevant things such as “I’m having a really bad day today” or “The train ride is terrible today“ -those kind of things. I think it’s better when everyone gets on and are friends, rather than fighting.\nKN: Yeah, basically people don’t like the politics. Different people have different opinions.\nMB: Let me tell you about my blog because for me the blog was really the starting point of everything. I was one of those frustrated people in my company. At the time, my ideas were kind of ignored by management, and I felt they were very good ideas. I came to a realisation that it wasn’t so much that I wanted them to do what I was saying, it was more that I wanted someone to acknowledge they were good ideas. And this was the whole reason I started blogging. I thought if I could just put my ideas out there, how I do something, how things are done, not only could the whole world contradict me and say “Actually, you might find this is a better way to do it” but I might actually help other people in those things that they might not know how to do themselves. I might even get people say “Thank you very much for helping me”. But it was more me just getting my stuff out there so I was no longer this frustrated individual that was being ignored. The aim of my blog is (I guess) interesting to some, it’s called tenbulls.co.uk. I think everyone goes through their Buddhist phase or as an interest in Buddhism and I like the Buddhist mantra of being good to everyone and everything -treat everything with respect, including animals. We all (to varying degrees) adhere to that as far as we can. Ten bulls are (I’m not really sure how to describe this) but there are some wooden prints, Buddhist prints, where they describe a man’s journey with a bull and how he tames the bull. And the ten bulls are the steps to enlightenment. So what I’ve got on my blog for the description is titled the steps to enlightenment with SQL Server and\u0026hellip;\nKN: A few days ago I went through those ten points.\nMB: I think Buddhist philosophy has a lot of\u0026hellip; even if you’re not religious in any way, shape or form, it’s a good lifestyle to have I think. I don’t always adhere to it but you know, I like to think I’m a good person.\nKN: OK, you mentioned something about respect for animals. Does this mean you’re a vegetarian?\nMB: This is where I said I don’t always adhere to it because I did, at\u0026hellip; I can’t remember how long ago it was now, but myself and my wife both did become vegetarians and we were vegetarians for over 10 years. The reason we stopped being vegetarians, we had two children and at the time our first child started to pick up really bad eating habits and we want our children to decide their future and their life, so the minute anything meat came on the table at school it was “Oh no, I’m not eating that because it’s meat”. Or questions about whether the food had “meat in it”. So, I made a conscious decision with the wife that ultimately they were more important than our personal ethics, and we can’t have them picking up our bad habits, so I actually forced myself back onto meat, which was probably one of the hardest things I’ve actually done. To eat that first piece of meat (it was a bacon sandwich) was really difficult because when you walk through a supermarket, as a vegetarian, you smell so much meat around the place, and it always used to make me feel really sick and I wasn’t sure how I was gonna react eating that first very first piece of meat.\nKN: And you know this data like “the meat consumption is growing and growing every year”?\nMB: Yeah, I think when my kids have grown up and moved out, there’s a very good chance I’ll become maybe not a vegetarian, maybe a pescatarian, so just eating fish. Its when somebody doesn’t eat red meat or chicken but they eat fish and seafood. You’re getting a lot of protein that way. I also found it’s very difficult (certainly in England) to be a vegetarian because (certainly from a fast food’s perspective) to just go somewhere and get some food quickly - it’s all meat.\nKN: Because it’s very cheap, unfortunately. OK, let’s go back to your company. You were doing your stuff across the whole of UK or\u0026hellip;?\nMB: I will go wherever somebody wants me. Obviously if I do travel into Europe to do some work which very rarely happens, it would obviously have to be financially viable for me to do that but yes, my work is really focused in the UK. And primarily I do most of my work in London. I work with a lot of financial companies so most of it is in London.\nKN: OK, London is not very close to you\u0026hellip;\nMB: It’s not too bad, though. Every day, I probably spend about an hour and 20 minutes going in there so it’s time to read a book or\u0026hellip; One direction.\nKN: And which one was first, your blog or SQLSaturday in Cambridge?\nMB: My blog. I think I started my blog in 2009 and the first SQLSaturday in Cambridge, if I remember correctly was 2012.\nKN: OK, so this is the 5th or 6th edition?\nMB: I think it’s\u0026hellip; we had one year where we couldn’t run due to personal circumstances, so I think it’s the 5th.\nKN: How big is the team that prepares this event?\nMB: That’s a very good question. At the moment, I primarily do 90% of the work, and my wife also does a substantial amount of work and that probably makes up the other 5–7%. And this year, we’ve got SharePoint MVP called Paul Hunt, who’s joined the team and I’m hoping\u0026hellip; and he’s doing a good job so far, I’m hoping he’s gonna become a really key figure from the SharePoint front and we can both work well together. He’s doing a cracking job so far today, leading the volunteer team at very short notice, so yeah. I think my biggest problem with this event has always been the delegation of tasks. I’ve had people in the past doing stuff and it’s always been very difficult, partly because I’m very protective about the stuff I do, because I wanted them to a certain level, exactly how I want it. And I think one of my weaknesses as a person is that I’m expecting much from people as well. I think that’s my biggest challenge going forward. To try and make this event not just seen as “Oh, this is Mark Broadbent’s event,” I want it to be seen as an event in its own right. And the day will come when I don’t want to do this anymore, because it does consume a lot of my life and I want it to still keep running. Even if it becomes a different event, like some time in the future SQL Server and SharePoint in their current format are not gonna be SQL Server and SharePoint -I personally believe. So I would like the event (whatever it’s called -and I know there’s Azure Saturdays starting now already) to continue without me.\nKN: So, because of those things, are you a perfectionist?\nMB: I really am. I’ll give you one example. I noticed I check things over and over again until I’m 100% happy. But I noticed that one of our sponsors in one place in our literature had an “i” of their name missing in one of the agendas that I had sent out. I was really annoyed about that since we had already instructed printers to start printing our schedules. So I said to the printer “Stop, we’re gonna get this “i” in.” and they did it for me, but lo and behold, I forgot to update my own material so yesterday when I sent out the electronic brochure to everyone, it had that missing “i“. That really annoyed me. I’m too much of a perfectionist but as DBAs, I always say we need to be pessimistic (that the worst could happen). We have to expect the worst and you have to have pride in what you do. It does slow you down, but at the end of the day, problems do not happen as often and your services will crash less.. If they crash, you’ll be prepared. You’ll know exactly what to do, you’ll have your disaster plan ready, so it does help with those kind of things. Plan B and also plan C.\nKN: How long, or from which version, have you been working with SQL Server?\nMB: I believe I started in 1997. SQL Server 6.5. And either 1998 or 1999 was my first clustered experience. So it was SQL Server 6.5 running on a Windows NT 4.0 cluster. The NT 4.0 back then was called Wolfpack, the NT 4.0 Cluster was called Wolfpack. The whole reason my career kind of veered off into an HADR avenue was my experience there, and it was because I was exposed to a serious disaster of the cluster. We had, it was the financial management at the time, and I made no secrets that nobody was allowed to go in to the server room at all without my permission. Anyway, one evening, I had the financial manager coming up to me and say “Going to the pub now, you’re welcome to join us but by the way, we’ve just had a vendor install some software on this server and I hadn’t noticed them go in to the server room. The whole cluster died when they were down at the pub. So that was a nightmare because I didn’t know enough about how to recover the thing at that stage. I spent a very bad weekend thinking that I’m gonna have to just hand in my notice, just to resign from the company because I couldn’t bring their system back online, or I didn’t think I could. Luckily, by Monday morning it was up and running. So from that experience, I told myself I will never ever put myself in a situation where I don’t know what to do.\nKN: And what kind of specific tools do you use in your work?\nMB: At the moment, it’s primarily SSIS, which for me is good because before SQL Server 2005 came out SSIS was introduced (before that we had DTS). I’d used DTS a lot and I’ve also been exposed to SSIS periodically since 2005. So every now and again, I’d need to go and write a package here and there and never touch it again for another six months. It got to a stage with me that I felt that I never really truly learned SSIS, so being exposed to it day-in day-out now is really good for me, cause even though I know SSIS still doesn’t have enough love from Microsoft at the moment, I still think in many businesses, SSIS is gonna be around for a very long time. And I think there are a few things to come in in SSIS that people potentially may like. But we will see how that goes!.\nKN: Do you use some tool to compare the meta data between versions, between the code\u0026hellip;, from SSIS perspective?\nMB: No. I had my first exposure with BIML recently, which I think in our company would be a godsend. But the problem is getting companies to accept software and sadly, they don’t like to try new things. So at the moment any code comparison tool for that, and it would be good to have something.\nKN: As a speaker, how do you prepare yourself for your speech, for a session?\nMB: That’s a very good question. When I first started, I used to put an awful lot of energy into it, I think if you are not confident about your speaking ability (even now I look at myself and I don’t feel like I’m a good speaker) you can still get through a session and produce some good material -but I’m always my biggest critic.\nKN: Because you’re a perfectionist.\nMB: Exactly. So I’ve done sessions and people have come up to me and said “great session” and I think they’re actually taking the mickey. I think they’re just joking -that’s what I’m saying to myself. But how I used to do has changed. Years ago, when I worked in a data centre (when I first started speaking) I would actually go into the server room and I would go through the entire session speaking to servers out loud, which was interesting because the very first session I did, I am almost shouting at the audience! Obviously when you’re trying to project your voice with servers humming in the background, you start getting very loud. So that was something I had to try and be careful of, not to shout too loudly, especially when I’m on a microphone. As years went on, I probably do less and less rehearsal as such, but what I would always try and do is this. I would try and run through a session perfectly once and have it recorded. And this is my top tip: I would have it recorded (because I hate listening to my own voice as I’m speaking) so I don’t have to keep listening to my own voice in real time. When I’ve got this really good session, you can piece it together, just in sound bites, but then when you play it along, it’s now a perfect session. A Dictaphone is really good for this and then you can rearrange it all so you end up with a perfect session. Now I would just replay the session over and over and listen to it. I’ve got a fairly good memory. I won’t say photographic memory, but I’ve got a fairly good memory, and I’ll just play it over and over and just watch the slides as I’m listening. By the time I’m ready for my session, I’ve pretty much learned the session so that I’m not having to stand up and practice speaking at all. I’m just listening to myself over again. Obviously, demos have to be practiced as well. So, while it all works, I think in reality it could be improved by maybe one or two rehearsals as well. But you know, I’ve got so little time these days that that’s partially why “I cheat”.\nKN: Let’s talk about the MVP. How long \u0026hellip;?\nMB: So I think I was first awarded in January 2016, and it became almost a running joke for a lot of speakers who are friends of mine that I didn’t get awarded (I’d been speaking Internationally since 2010). And one of the reasons why I hadn’t got awarded up to that time was because I think there was a freeze in the UK on new MVPs so even though you were potentially doing loads of things in the community and maybe were even at a certain technical level, there was perhaps nothing they (MSUK) could do about it because they obviously have a budget to have only a certain number of MVPs -but Microsoft now seemed to have turned that on the head in the UK and more and more new people are getting MVPs, so I suspect I was one of the first batch of “new” MVPs. And it’s funny because the year that I did get awarded it, I’d had 20 nominations all at the same time. I don’t what the record is but for me at least that felt like a lot. But I think my biggest problem with the getting the MVP award -is that now they could take it away! So, it’s always one of these things that I don’t take too seriously, and I’ll be entirely honest with you, when I got my MCM, the Certified Master certification, I was literally punching the air in delight and jumping around the house, but when I got the MVP, even though I was delighted to get it and pleased I have it, it wasn’t the same mode of elation. Because there’s an awful lot of people out in the community contributing to its success and it’s very easy to not get appreciation or feel like you are getting recognition for these things. What I really don’t like is when people get awarded the MVP and then they start acting in a different way, like they’re super-special. I think all of us are the same, even people that just attend events, we’re all the same, we’re all on the same level, some of us may spend a bit more time maybe doing a bit more community stuff, some of us may spend more time doing technical stuff but we shouldn’t really feel any different or special because of that.\nKN: But still, it’s an award. If you are MVP, you can feel like a special guy.\nMB: While we’re all on this broadcast, what I would really love to happen with the MVP programme (and it’s something I have mentioned when we’ve had these discussions in MVP discussion) is I would really love there to be like a\u0026hellip; If financial things were a consideration for the numbers of MVPs, I would like there to be something extra that maybe is just a stepping stone into the MVP. Something like: “We recognise you as being technically gifted and contributing your gifts to the community, and you’re kind of almost there, so you’re on this initial stepping stone programme. Now you have insight into all our up-and-coming technologies and you can contribute to the community better. Because of this, you have more connections, more leverage with Microsoft and more say what happens. Keep going!” And from that, there’d be a lot more people, a lot more community, and then the next step is just saying “Well done, you’ve now achieved the level we want”. So, I think there needs to be something like that, rather than what appear to some as random awards. Sometimes, It really does almost seem like a random selection of people from the outside.\nKN: Do you think that this is random currently?\nMB: It’s not random, I’m being very unfair in how I’m talking about this. It’s not random but I think\u0026hellip; I’ll give you an example: if nobody had ever voted for me or nominated me, doesn’t mean I almost definitely wouldn’t get the award. Usually, someone must nominate you (but perhaps not in every single time). You could (if you are that much of an egotist) nominate yourself though! I do try to nominate people. If I’ve seen people do a lot on the community, I will nominate people. I mean, they obviously have to be up to a certain technical level as well but I will nominate people because I think it’s important that people get recognised. Even just the nomination for some people is enough, and I will always explain how it works as well. You’re not guaranteed anything, you’ll not be notified if you don’t get the award. Don’t be upset about it, I keep trying to nominate and see what happens down the line, so just keep doing what you do. But I don’t think it’s important that people\u0026hellip; people should not work towards the MVP, they should see it as like a fringe benefit almost, I think.\nKN: Just do what you want to do and that’s it, focus on that. Do you remember your MCM exam?\nMB: I certainly do, yeah. It was the second strain of the MCM exams. There was the version\u0026hellip; Microsoft initially had the Ranger programme, where people would have to physically be on Microsoft premises in Redmond, and they would spend, if I remember correctly, it was either a week or two weeks there, and it cost about £10,000-£20,000 do it. Very, very restrictive, you had to get your company to sponsor you to do that. And even back then, I was thinking “I’m gonna save that £10,000 to do this. It’s gonna take me a few years to make me save that up, but I’m gonna do that.” And then they changed the name of the Ranger program into the MCM, and they still had that for a time where you had to visit for a week or two weeks and pay the money. And then, I think it was like 2010–2011, and I really became aware of the MCM through people like Brent Ozar who were blogging about it some of the time, and then they changed it so that all you needed to do, was you had to do an almost 4-hour knowledge exam, which was a bit like we’re all used to the Microsoft multiple-choice or multiple-guess exams but it was a little bit more intense than that. The questions were a lot cleverer. But even just sitting an exam for 4 hours in itself, regardless of the fact it’s multiple choice, that was quite a hard exam to do. I passed my knowledge exam in 2012 I think. And then we also had what’s known as the Lab exam, where you physically have to dial in two Microsoft’s premises and you connect to their servers and the lab environment, and I can describe it as this: imagine your very worst day at a company when absolutely everything that could go wrong, goes wrong. And this is exactly what we had. We had scenarios, if I remember, there were 12 questions. Doesn’t actually sound very much in itself but there were 12 questions, you could spend half an hour to an hour on just one of those. So what you have to do is kind of look at these things and think “How long am I gonna spend on this thing” and I don’t think I’ll break an NDA saying this, but on the very first question, it was one of these gotchas where to arrive at the solution, it would take you\u0026hellip; and it was a backup and restore, to actually get the right answer would take you at least 30 minutes, if not longer. Or could kill you for about an hour. And now you’ve got 3 hours left to do the rest of it. So that was one of those little gotchas. It was one of the hardest things I’ve ever done, but it was also one of the best I think. You also had a camera that was pointing on you and recording you, so they could actually see your movements. You couldn’t see who it was but they could see you and make sure you weren’t cheating or looking in books or asking your best friend. Maybe Brent Ozar, asking him for the answers! You were allowed 5 minute breaks to just go off and get a cup of coffee every now and again.\nKN: And check the book again.\nMB: Yeah, make sure your books were downstairs! And all the paperwork as well! You’re allowed to\u0026hellip; I can’t remember now if you actually were allowed to write anything down, because you certainly weren’t allowed to take any paper away. But I’ll tell you what I did. What I did is write after the exam, after I’d sat the Lab, I actually wrote from my memory, just wrote everything down I remembered. And I didn’t pass first time. Probably most MCMs did not pass first time cause it was that hard. But obviously I would then go and research things that I was very weak in and then finally got the MCM. When I found out, it was at 5am and I had to go to work two hours later, having been up most of the night. I had to go to work but I was literally punching the air, tweeting away how happy I was. There’s quite a nice photo where I have some of my friends who also took the same journey as me sat together in a restaurant. You’ll have heard of Edwin Sarmiento probably, who’s quite well known in the SQL and HA community. And he was on the SQL Skills course with me and there’s a guy called Shawn Gallardy as well, Ryan Rhineheart as well was with us. And a couple of other people who became friends. But there’s a shot of us at the table, and everyone you can see in the shot got their MCM before Microsoft cancelled it.\nKN: Congratulations!\nMB: Shame they retired it but there you go.\nKN: What is your hobby? Do you have any hobbies?\nMB: I used to have hobbies, when I had a life. I’ve got two young children and I guess any time that I’m not doing anything technical or business-oriented, or event planning, speaking, I’m generally just spending with them. There was a time I could tell you lots of different hobbies I used to do which I struggle to remember now but one of the things I’m still quite passionate about is film. When I was a lot younger, I used to record every single film on telly and I had a film book, like a thesaurus almost of films and reviews of films. So, what I would do I would watch a film and I’d give my own little review. And I was watching an awful lot of films! Every day I was watching about 3–4 films. I’m a big fan of black and white films. My favourite film of all time is Night of the Hunter, starring Robert Mitchum. It’s quite a famous black and white film. Robert Mitchum plays a “preacher” who is actually a prisoner, but then he’s released. He’s a really bad guy, a token bad guy. He finds a widowed woman who has two small children and he basically marries her just to get her money and then he kills her. But her two children try to escape him but he comes after them because they actually buried the mother’s money to keep it safe, and he doesn’t know where it is. So, he’s gotta get these children and he’s gonna kill them when he finds them. One of the visual scenes in the film I really love (it’s a rather chilling image) is shown as the kids are escaping by boat. The way he killed the mother was (if I remember) by tying her up in the car and\u0026hellip; it’s obviously an old -fashioned car, one of these open-top things, and he drove it and her into a lake. So there’s this amazingly horrific scene where these two children are fleeing Robert Mitchum, the preacher, and are rowing in this rowing boat over the water to escape -and it’s a really sunny day. The sun’s shining through the water and they row over the silhouette of their mother in this car, dead and tied up and they don’t notice it. It’s really a horrific thing but it’s also a beautiful visual to see as well. I really love that film.\nKN: That sounds really interesting. I must watch that film. But probably these days you don’t have time\u0026hellip;\nMB: Not as much. When I do less speaking and less event running\u0026hellip;\nKN: And have less children, I guess it’s not possible now. So how about your life-work balance, how is that you can\u0026hellip;\nMB: My children would tell you that I do far too much work. I find it quite hard really to switch off from work and my life but if I have definitely turned off from work, don’t try calling me, don’t try emailing me cause I’m not gonna reply to you. But generally, I spend too much time doing work and I do need to maybe get more work-life balance and spend more time with the kids I think. Something I’m really working on. Last year for instance, I did so much travelling, my wife said to me\u0026hellip; in fact she said to me over social media, funnily enough (that shows you how often I was away)! She said to me over social media: “Looking forward to next weekend. It’s the first weekend in 3 months that my husband will be around”. I was doing so many events and it also cost a lot of money. I ask why am I doing all this, what benefit do I have from doing it. But it’s almost a thrill when you speak, it becomes an addiction. It’s very hard to keep up.\nKN: So you’re still looking for this perfect way\u0026hellip;\nMB: This comes back down to more people, more volunteers getting involved and them all taking responsibility for the tasks I may have done in the past.\nKN: How old are your children?\nMB: 8 and 11. It’s an interesting age.\nKN: I think it will be more interesting in the next 4 years.\nMB: Yeah, and I’m not really looking forward to that, to be honest. They’re growing up far too quickly now. That’s one reason I wanna be around more just to\u0026hellip;\nKN: I can imagine. I have the same. I have two daughters, age 1 and 4 so also I’m working in London but the time to travel is one hour one way and one hour back. And also after work, I want to do something after my job, for example learning something new or checking something or just edit the podcast or just put some old post on my blog and I just put more work on my back.\nMB: I find blogging really time-consuming. I never remember taking so long before. I had blog posts very recently, must’ve taken me 6 months to write because I kept writing and then it sounded rubbish. I kept going back to it and then adding this. Then came the day where I thought I had to get this pushed out so I spent like 5 hours tweaking it. And I pushed it out eventually and it became\u0026hellip; I can’t even remember what it was now but it was something really basic on SQL Server. But it was just\u0026hellip; I don’t know. To do with\u0026hellip; I think it was doing financial count or something like that. No, it was conditional count. And I pushed it out and it was one of my biggest ever blog hits that I had ever had. Hundreds of hits within the first day, second day, third day, and I’m like “What’s going on, this is crazy!“\nKN: So you found a very hot topic.\nMB: Well, I don’t know whether something happened or\u0026hellip; Sometimes when well-known people repost you or whatever, you do get a lot of hits. And I wasn’t sure if it was related to that or what but it was strange. That really showed me that you shouldn’t be too clever on the blog, just post something that’s relevant to you. Keep it simple, just get something out there. My biggest problem was always that I don’t want people to think\u0026hellip; like if you publish “How to install SQL Server” for instance, you may think that’s gonna make me look like I don’t really know what I’m doing because “Who the hell doesn’t know how to install SQL Server?” So you have that mentality. But then I have another blog post I did, it was “How to install SharePoint 2013 on the Evaluation Edition of Windows,” if I remember correctly. At the time, you could only download the Evaluation RTM edition of Windows, you could not download the Evaluation edition with Service Pack 1.\nKN: I remember the bug, I spent a lot of time to find the solution.\nMB: This blog post, I’d spend ages in, I found lots of different sources but none of them actually tied up the answer neatly, concisely. And step by step, and it all worked. So, I basically look to all these different sources and then at myself: “Let’s just get it how I would want to.” That blog post is probably my most successful blog posts ever. And it’s SharePoint.\nKN: I must double-check that.\nMB: So anyway, I’ve got a 2016 version that I’m gonna have to do to keep the hits going!\nDamian, so maybe you will start with your questions because I don’t have a lot of them currently.\nDamian Widera: OK, I’ve got one more question about MCM. Maybe it was not yet posted, the question, so Mark, do you think it was a good move from Microsoft that this programme was killed like 3 or 4 years ago?\nMB: That’s a very, very good question. No, I don’t. I think every MCM that sat the MCM believes the same thing as well. It’s funny because there’s a lot of people out there, well-known figures, I get this\u0026hellip; a lot of my friends on Facebook who are SQL Server people, they’re quite critical of the MCM saying bad things like “Oh yeah, that’s why I’ll never sat the MCM.” It sounds to me almost like they want to justify in their heads why they don’t have one because it’s one of these silly, frivolous exams that Microsoft did. And my response to them is always the same, and I say “I never hear of an MCM saying that it was a waste of their time to do it.” And that maybe if you’d done it, you would realise what value it has. What you’ll get is a lot of MCMs who did sit the exam. They’re very passionate about the exam and I know fellow MCMs who I respect highly. I don’t actually know them personally but I respect them because I know what they went through to get the award. It wasn’t one of these things where you could do a transcender exam and then pass the test. It was a lot of work, a lot of white papers. And I think what Microsoft have done, they’ve almost put the MCM in\u0026hellip; I’m trying to find of the right term but it’s almost one of these fantasy things now, where there’s these people called MCMs and there’s no way for other people to ever get in. So, I see other MCMs almost as like these fantasy figures almost. I think Microsoft really need to give all these top-quality SQL people some way to demonstrate to others how good they really are. I don’t think it’s fair that they can’t do that. There has been a lot of talk, you guys will probably be aware of the Microsoft professional programme where there’s a data science course that you can do and get a\u0026hellip; I think it takes a year to do this. And I’ve heard really good things about this. I’ve heard Thomas LaRock talking about it on his blog. And there’s talk about Microsoft doing something similar for DBAs, for maybe an MCM-level certification. I think something like that might be good but no, I think it’s a bad idea Microsoft cancelled and hopefully one day they will reconsider. But the story always goes that the reason it was cancelled was because Microsoft Learning weren’t making enough money from the programme. And that was the bottom line.\nDW: Oh, I was expecting that something will be\u0026hellip; if the programme is cancelled then something new will come into the light but for four years I haven’t seen anything like that, so that’s why I’m very surprised because it was a really valuable programme in terms of knowledge that you had to possess and well\u0026hellip;\nMB: I think one of the problems is that our world is really changing so there will come a time very quickly that traditional DBAs are not gonna exist. We’re gonna be expected to do a lot of things in cloud, a lot of things in other technologies, Linux is a classic example. And how on earth are Microsoft gonna do an MCM-level test for all of those things. You’d be sitting like 50 exams, 50 lab tests to get anywhere close to that. Maybe they need to have something that is a lot of harder to pass than the current MCP exams.\nDW: Yes, that’s right. And you’ve been talking about DBA. And I also observe that the role of DBAs is getting less and less important because there are a lot of features in SQL Server that try to eliminate the DBAs, like for example auto tuning of queries and smart backup. For some companies could be a good idea not to employ a DBA because you can have backup into cloud. Do you think we’re going in the right direction?\nMB: In terms of companies’ perception for the need for DBAs no, I don’t think we’re going in the right direction. In terms of how Microsoft are trying to improve the experience on their products, yeah, I think they are. But I think what they need to be really doing is making the experience better for the people administrating the product, rather than maybe trying to eradicate, and I’m not sure they’re trying to eradicate, I just think it’s a consequence of what they’re doing. But I think having certain people like DBAs positions under threat because of all these technologies is actually a false thing for companies to do because in my experience, DBAs are definitely not regarded as highly as they were at once stage, or not as much as a requirement for companies. If you asked a company if they want a data scientist or a DBA, the data scientist will get the job. That said, there is nobody more important than a DBA in a company when there is a fault on the SQL Server. Suddenly, the DBA becomes the most important person in the room and all the managers are suddenly taking their interest. I think maybe more could be done from\u0026hellip; maybe Microsoft even, to educate companies on what their requirements really should be. In order for us to support you, we need to have a DBA looking after this product, the deployment, the settings and everything. And they need to be following maybe our best practices. Otherwise, you’re unsupported. And that way, companies won’t think they don’t need DBAs, they’ll think “We do need the DBA but maybe we don’t need as many DBAs.” So it’s interesting the way things are going. Time will tell.\nDW: Because like a few years ago, when the DBA was really an important person, for example Paul Randal has an excellent blog, and many other people have excellent blogs with great examples what to do in case of corruption or problems, or failures. And right now, DBAs are not going to be so useful. So in case of a simple problem, there will be no people around who can help you with SQL Server. That’s my concern.\nMB: We all know that the minute that Paul Randal becomes a data scientist, it is the day that we need to stop doing DBA work. But we do have a scenario where\u0026hellip; we’ve got this new thing now, in 2016 -I think it came in, where we have the multiple temp files that we set up on installation. But we still have the scenario where there’s gonna be lots of legacy settings where you have to be able to administrate the whole environment to a certain level and standard. And just having newer products do things automatically, it may do it automatically but it may be wrong. It may be “right” on that server that you’ve automatically installed SQL Server with four temp files. But in six months’ time, the plan is to put more CPU power (cores) in the server and suddenly the tempdb thing may be wrong for this hardware. There’s an awful lot of things that cannot be automated or set with a “one size fits all”, and the classic is autogrowth settings. I personally don’t like autogrowth at all. I like to fill it out as far as I humanly need. And I maybe allow it a few extra growths because the minute you get an autogrowth, you know you are in serious space problems. And you know that you need to speak to someone about storage, getting more storage. But SQL Server can’t make those decisions for you. Maybe this is where machine learning comes in and sooner or later we’re gonna come to a time that the installer will know your habits and how you do stuff, and how you deployed other servers and make decisions. When that happens, we’re definitely out of the job but I can’t see that happening for a while.\nKN: For example like for Query Store, this is I think the example where DBAs don’t have to spend a lot of time analysing the indexes, because the Query Store and machine learning and those kind of things, future can do that for you. And I saw some articles about it and very likely it will do better than you. Maybe not in every case but\u0026hellip;\nMB: Imagine a situation with Query Store where we have… and let’s say everything was automated, so it would keep swapping the plan to how it really needs to be automatically. Imagine a situation where every plan that’s created is wrong, based upon the distribution of data. So every time the plan created is wrong, the Query Store is then automatically trying to give you the right plan. You’ll get in a situation where it’s constantly changing, so you have got a brand new bottleneck. There is no right answer here at all and you need to say “Look, just leave that alone, we cannot win whatever plan you choose.” So I think by a lot of the automation that happens, we are potentially gonna get new problems to solve.\nKN: It’s possible but I’m thinking about a situation where it can help. I’m not talking about how everything could be automated. I think it won’t happen so fast that DBAs won’t be useful.\nDW: Yes, but we are definitely going in the wrong direction maybe with these DBAs.\nKN: But this is another story. This is a story for another long conversation I think.\nDW: So maybe some other question. Other topic. Mark, how did you start your journey with SQL Server. I hope this question hasn’t been asked.\nMB: That’s an excellent question. I was a Novell man and I did pretty much everything in this company, from desktop support to Novell Server support. I loved Novell as well. And I was working for a Saudi Arabian company at the time and I have a different perspective on pirate software than other people because I think pirate software can actually lead into positive things. One of our managers at the time, I’m not gonna name the company because of the pirate software, went to Saudi Arabia, came back with a CD of lots of different bits of software on it. One of which was NT 4.0 and back in that time, finding jobs with different skill sets was very hard to move into, unless you had experience. So I actually tried out NT 4.0, got very good with it, and I loved the experience as well. Then I very quickly got a job working for another company, doing the same sort of stuff but in purely Microsoft technology, so NT 4.0 Server, NT 4.0 Workstation, and there was also backend software such as Exchange server, which I was also exposed to, but also a piece of software called SQL Server. And this SQL Server installation used to manage all the corporate documents at the time. There was a product called PCDOCS Open and it would store the images of the corporate documentation within SQL Server. That was my first piece of exposure to SQL Server and I very soon realised that I wanted to focus more on SQL Server. Back at that time, SQL Server 7 was on its way out and there was a beta training course that came out, and my boss said “Put yourself on a training course for SQL Server,” so while he was expecting myself to book on a 6.5 training, instead I booked myself on SQL 7 beta training course. All of a sudden, I became very good at SQL Server and very marketable to other companies, and rest is history really. That’s how I moved on. It was a very good move.\nKN: How often are you a speaker on different conferences?\nMB: Last year was probably my peak, and I must’ve spoken at a conference maybe every other week. So last year I think I went to America four times to speak. Europe, anything from 6 to 10 times. A lot of time. This year a lot less. I’ve probably done about 4 conferences so far. Probably will do about 3 more before the year’s out.\nKN: Have you ever been to Poland?\nMB: I haven’t, no. You guys have a couple of conferences there, don’t you?\nKN: Yeah, we have a very big conference, SQLDay. You must visit us!\nMB: Yes, I’d love to come some time.\nDW: I have one question about these conferences and speeches. I bet you really like it Mark, talking to people, sharing knowledge and give experience from other people?\nMB: Erm, there’s something about it that I definitely love. Seeing brand new countries and meeting people, and seeing people you can potentially make a difference to someone is great. There are bits about speaking at other places that I’m not so keen on. It’s being away from the family. We had this discussion earlier about\u0026hellip; I think it was offline, where we were talking about work-life balance. It’s something that I generally get wrong at the moment, so I do need to try and focus a little bit more on spending more time with the family. But I obviously get a buzz out of it, otherwise I wouldn’t do it.\nDW: Maybe one more thing. How do you prepare, if you know there will be a speech and 500 people in the room?\nMB: We talked earlier on this. I will listen to a recording of myself doing the session and I’ll listen to it over and over again. But another thing that I do is (and I learnt it at my very first presentation) I felt the nerves coming and I told myself “This is good, use it to help focus”. So when I feel nerves coming, I would always tell myself that and instruct my body to kind of go with it, take the nerves, take the adrenaline, use it. And whenever I tell myself that, the nerves and any shaking I ever did have will subside. I’m fairly good though now, to be honest. I don’t get as much nerves as I would really like, because I’d like to have adrenaline. I think it’s a good thing to have, always to try and stay on the edge of just falling apart or delivering the most amazing session.\nKN: Recently, Rob Sewell posted a very good article about it. He mentioned that you should be nervous before your session, it’s a good thing.\nMB: For me, I’ve done so many presentations, that’s very hard for me to get those nerves. I almost don’t like not getting the nerves. Another thing I have found very successful for me is I always like to have a beer. So, I think I’m quite unique in this. I’ve done it a few times and I’ve had really good feedback when I’ve done it. I think it just relaxes me a little bit. And I also think it’s quite awesome as well. You’re giving a session and you’re obviously going through a stressful period, stressful hour but then you get a beer and you can chill out and you can enjoy it. I’ve actually found that a very good experience. I’ve come out of sessions and I’ve done before and I’d hear “Wow, great session,” cos I’ve had a beer. So that’s another top tip!\nDW: Definitely worth checking, maybe SQLSaturday next Saturday in Poland, I will have to check it if it’s possible to talk and drink at the same time.\nKN: You are a good professional at your topic, in terms of SQL Server and probably a couple more things. The question is, are you still teaching something new during conferences?\nMB: Every now and again, I’ll add in a new topic, a new session. I don’t necessarily tend to talk about a lot of the old legacy stuff. It’s all sometimes about up-and-coming technologies. Microsoft do like you to push their up-and-coming technologies. Me, it’s more from an interest perspective. And I’d be very neutral in terms of how I will explain a topic. For instance, I’m one of the biggest critics of Stretch Database. If you want to ask me about it sometime\u0026hellip; But that’s another story. So I do try to talk about a lot of up-and-coming things. Microsoft have asked me recently to talk about GDPR, and I don’t know too much about it yet, so I have to start learning.\nKN: It will be a very hot topic. We have time till mid-May\u0026hellip;?\nMB: Yes, a few months.\nKN: Which of your achievements are you satisfied with the most?\nMB: In life or technical? I think I would be an idiot not to say my children. I’m not a very emotional guy, I bottle my emotions. From an anger perspective, I could be very annoyed at something but it would take a lot of me building that annoyance up to actually demonstrate it. I may very casually say I’m annoyed but you’ll never be frightened by me losing my temper! The last time I cried, I think it was the birth of my son and the time before that was the birth of my daughter. It kind of surprised me actually. I was overwhelmed with emotion when my daughter was born. I had been quite stressed for a long time because it was a really long birth and I walked out of the room and it was all over. I went quietly into the bathroom and literally just cried a little bit. It took a lot out of me but you won’t see me crying anywhere else I’m afraid.\nKN: Still, I think you’re an emotional guy because for my daughters, I didn’t cry. I should say that I’m not an emotional guy. But probably you are.\nMB: Maybe I am. I have found, I’ll be honest (since the children) I have found things really get me that didn’t used to. Like I can watch films now and I can feel the tears coming in the cinema, and I’m saying to myself “what is wrong with you?!” Certainly, before the children that never, never, never happened. So, I think something has flipped in my head.\nKN: Which is good I think.\nMB: Maybe.\nKN: Do you have any animals at home?\nMB: If you had asked that a while ago, I would have probably said one cat. We lost a cat last year due to old age and now we’ve got a dog in the last few months, and we’ve got two brand new cats as well. So, the house is a menagerie of children pets and the wife of course.\nKN: What are your daily habits?\nMB: Get up, go to work, go to Starbucks and come home, watch a film with the wife, go to bed. That’s literally it. Oh, and also every day I will probably have gin and tonic in the evening as well. That’s a really nasty habit that’s formed over the last six months. Every single day, I’ll generally have a gin and tonic. And it’s costing me a lot of money.\nDW: So the last question, if you can hear me, would be: what would you say to people who would like to start studying IT science? Where should they go? What specialisation should they take in your opinion? In terms of data platforms let’s say, it could be some\u0026hellip;\nMB: I would focus on the official training courses that Microsoft release. We mentioned the Microsoft Professional Program for Data Science. That’s a really good thing to do, and it’s also aimed at anything from beginners to intermediate people. I’m a great believer in books as well. I’m also a great believer in finding the right book for you. Many, many years ago I was trying to learn C#, and I went to a book shop, obviously book shops don’t exist as much as they used to back then. I went to a book shop and I had about 20 C# books just stacked on the floor, one after the other, and I sat down on a bench and went through every single one, and just skimmed through it, read maybe a paragraph or two, and I would put that one down, saying “no, that sounds rubbish the way it’s written” or “it’s just not digesting in my head.” And finally, it was nearly the last book that I had actually got to and suddenly it was beautifully written. Everything I understood that it said. And I bought that book. I’d read it front to back, probably several times, and learned.\nKN: Yeah, but it’s not very easy to sit and review a book in the shop and decide which one you want.\nMB: It’s not but you don’t have to spend a lot of time. You know from one paragraph whether this author is gonna really do it for you or not. So I think that’s it, just find the right book, find the right materials. I also subscribe a lot to Udemy as well, which provide a lot of unknown courses. I find some of the courses are mixed but just get as much stuff as possible. You’ve got Pluralsight as well that you can generally get through your company, get a lot of subscriptions for free from events and user groups which give them away. There’s a lot of stuff out there.\nKN: Currently there’s a lot of information in the Internet.\nMB: There’s too much really.\nDW: But I think the most important thing is that we have to learn every day, and we cannot stop learning because the technologies are changing so quickly and new things come into the light so we cannot stop learning. And that’s what I think is important.\nMB: There’s always that younger guy who’s better than you, who’s left college, he’s faster than you, he’ll make more effort than you. When I was younger, I could stay up to 4am learning something, I would get to bed for 2 or 3 hours, go into work and I’d be tired but I could do it. Now, you would not even get me going past (apart from organising my event) midnight to do learning. I want my bed, I want my sleep, I need it. So, yes, we have got to stay ahead of the curve.\nKN: We are not young guys anymore.\nMB: Speak for yourself!\nKN: We need some sleep. OK, at the end of our conversation, tell us where we can find you.\nMB: You can find me through my blog, which is tenbulls.co.uk. I do lurk around Twitter under the name of @retracement. I maybe don’t do as much Twitter as I used to do but I am there. Potentially, you could send me a LinkedIn message although I don’t tend to look at that very often or connect with people very often. I just never get around to doing that. But the main thing is if you do need to get hold of me, send me a tweet, even if it’s just to ask a simple question about something, I’ll do my very best to accommodate and communicate, or even point you to someone who can.\nKN: And obviously people can find you at SQLSaturday Cambridge.\nMB: Yeah, they’ll see me lurking around. If you come to SQLSaturday Cambridge, you’ll definitely see me lurking around. I’m the guy who’s looking stressed and unhappy.\nKN: Oh, you don’t. OK, thank you very much.\nMB: Thank you for having me, it’s been a pleasure.\nDW: Thank you, have a great day in Cambridge!\nMB: You too, cheers, thank you!\nUseful links: Mark Broadbent’s website: Blog\nMark Broadbent’s Twitter: @retracement\n","date":"2017-10-26T23:17:07Z","image":"/uploads/2017/10/IMG_6595.jpg","permalink":"/2017/10/asf-004-mark-broadbent-interview/","title":"ASF 004: Mark Broadbent interview"},{"content":"Hello SQL Folks. We all love USQL for its great extensibility and how it makes our life easier in some particular \u0026ldquo;area of data\u0026rdquo;.\nThere are a lot of things that have been updated this Summer. Complete list of fat can be found here.\nI will give you some practical example of them in one of mu further posts but today I would like to point you to the feature that is already available however it is not docummented. I would like to thank to Michael Rys to point out to this feature :).\nLet\u0026rsquo;s start with something you already know. USQL allows you to define a function written in C#. I have demostrated this a lot during my session in Copenhagen and SQLDay 2017 in Wrocław.\nI have a set of files in one one my ADL directory and read them by using a TSV extractor:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 DECLARE @in string = \u0026#34;/Data/Aircraft/2006ByMonth/{*}.tsv\u0026#34;; DECLARE @out string = \u0026#34;/Data/Aircraft/2006ByMonth/out/CSharpFunction.tsv\u0026#34;; @search = EXTRACT Month int, DayOfMonth int?, DayOfWeek int?, UniqueCarrier string, FlightNum int?, TailNum string, ActualElapsedTime int?, CRSElapsedTime int?, AirTime int?, ArrDelay int?, DepDelay int?, Origin string, Dest string, Distance int?, CarrierDelay int?, WeatherDelay int?, NASDelay int?, SecurityDelay int?, LateAircraftDelay int? FROM @in USING Extractors.Tsv(); Then in the second step I do a call to a function named \u0026ldquo;AddOneMonth\u0026rdquo;.\n1 2 @m = SELECT DISTINCT SQLSatDenmark.Udfs.AddOneMonth(Month) AS AddedMonth, Month FROM @search; The function generates a new column (AddedMonth) which is later send to the output file along with the existing Month column.\n1 2 3 OUTPUT @m TO @out USING Outputters.Csv(); Now let\u0026rsquo;s see how the function look like. It is really hard to imagine that something could be simpler than that. And not working well in all cases but the goal of the function is to show how to use the C# code in the USQL code.\n1 2 3 4 5 6 7 8 9 10 namespace SQLSatDenmark { public class Udfs { public static int AddOneMonth(int month) { return month + 1; } } } Now let\u0026rsquo;s go to the new thing - undocumented usage of inline functions. My function is pretty simple and I can imagine that function you will write could be as simple as mine but your functions will probably do something more useful. To simplyfy the coding process you could use inline function in your USQL script and not have to write any code in the C# file.\nThe could could look like this:\n1 2 3 4 DECLARE @in string = \u0026#34;/Data/Aircraft/2006ByMonth/{*}.tsv\u0026#34;; DECLARE @out string = \u0026#34;/Data/Aircraft/2006ByMonth/out/CSharpFunction.tsv\u0026#34;; DECLARE @func Func\u0026lt;int,int\u0026gt; = (s)=\u0026gt;{return s+1;}; Now I skip the EXTRACT part of the code as this has not changed. What has changed is how the @m recordset is prepared:\n1 2 @m = SELECT DISTINCT \u0026lt;strong\u0026gt;@func(Month) AS AddedMonth, Month FROM @search; How it works? This is just a regular inline function that you might probably know from C#. The function can be used only in this particular USQL script and cannot be referenced outside it. Hence we can call it a private function.\nIn my example the function takes \u0026ldquo;s\u0026rdquo; as parameter of the type integer and returns also integer value.\nIn general, the function can be vry complicated, for example take a look here:\n1 2 3 4 5 6 7 8 9 DECLARE @func Func \u0026lt;string,int?\u0026gt; = (s) =\u0026gt;{int i; var x = Int32.TryParse(s, out i); return x?(int?)i:null;}; OUTPUT ( SELECT @func(\u0026#34;12\u0026#34;) AS ParseStringToInt FROM ( VALUES(1)) AS T(X) ) TO \u0026#34;/output/ParseStringToInt.csv\u0026#34; USING Outputters.Csv(); Can we do more than this? Sure we can. You can have more than one function defined in the USQL script:\n1 2 3 4 5 6 7 8 9 10 11 DECLARE @stringFunc Func\u0026lt;string,string\u0026gt; = (s) =\u0026gt;{return \u0026#34;555\u0026#34;+s;}; DECLARE @func Func \u0026lt;string,int?\u0026gt; = (s) =\u0026gt;{int i; var x = Int32.TryParse(s, out i); return x?(int?)i:null;}; OUTPUT ( SELECT @func(\u0026#34;12\u0026#34;) AS ParseStringToInt, @stringFunc(\u0026#34;009\u0026#34;) AS StringFunc FROM ( VALUES(1)) AS T(X) ) TO \u0026#34;/output/ParseStringToInt.csv\u0026#34; USING Outputters.Csv(); Imagine we can do even more. Let me show you two other examples now.\nLet\u0026rsquo;s analyze the code. I have declared two functions. The @stringFunc function is responsible for concatenating two literals. The @func function tries to convert a string into an integer value. The \u0026ldquo;magic\u0026rdquo; happens in the SELECT statement where I use the @stringFunc function as a parameter to the @func function.\n1 2 3 4 5 6 7 8 9 10 DECLARE @stringFunc Func\u0026lt;string,string\u0026gt; = (s) =\u0026gt;{return \u0026#34;555\u0026#34;+s;}; DECLARE @func Func \u0026lt;string,int?\u0026gt; = (s) =\u0026gt;{int i; var x = Int32.TryParse(s, out i); return x?(int?)i:null;}; OUTPUT ( SELECT @func(@stringFunc(\u0026#34;009\u0026#34;)) AS ParseStringToInt FROM ( VALUES(1)) AS T(X) ) TO \u0026#34;/output/ParseStringToInt.csv\u0026#34; USING Outputters.Csv(); Let me now change the code a little bit. I have changed the definition of the @func function so it now calls @stringFunc function and processes it. The SELECT statement will be much more readable in that case as you might expect and the complexity of the call from the previous example is hidden in the function body.\n1 2 3 4 5 6 7 8 9 10 11 DECLARE @stringFunc Func\u0026lt;string,string\u0026gt; = (s) =\u0026gt;{return \u0026#34;555\u0026#34;+s;}; DECLARE @func Func \u0026lt;string,int?\u0026gt; = (s) =\u0026gt;{int i; var x = Int32.TryParse(@stringFunc(s), out i); return x?(int?)i:null;}; OUTPUT ( SELECT @func(\u0026#34;009\u0026#34;) AS ParseStringToInt FROM ( VALUES(1)) AS T(X) ) TO \u0026#34;/output/ParseStringToInt.csv\u0026#34; USING Outputters.Csv(); Enough for today\u0026hellip; Should we use inline functions? Hell, yes!!!!\nThey have all the advantages as they have in the C# programming\nCheers\nDamian\n","date":"2017-10-20T14:12:44Z","image":"/uploads/2017/10/P1277230.jpg","permalink":"/2017/10/functions-in-the-usql-the-hidden-gem-in-the-summer-2017-update/","title":"Functions in the USQL - the hidden gem in the Summer 2017 Update"},{"content":"It was a great weekend in Copenhagen. Kamil and I had the opportunity to give lectures during #656 SQL Saturday. I started in the morning talking about the U-SQL and Azure Data Lake (ADL). Everything went very smoothly but as always, it was good to check demos before the session. The ADL and U-SQL have new functionalities and some others were refreshed just two days earlier. Here you can find the details.\nMy second session was about Row Level Security and some practical usage aspects. I explained that it is and was possible to prepare a very similar solution on your own in all previous versions of SQL Server. We end up with a great discussion what are the benefits of RLS (Row Level Security) over the custom implementations. Take a look at the links below and check in the code/slide deck I share with you.\nKamil was talking about maintenance of the DB projects and Continuous Delivery using SSDT and Power Shell. I love the topic because this is what I do all day long and day by day. You could hear many good practical examples. He went through an SSDT in Visual Studio 2017 briefly, has explained how to cope with some common issues with the tool, focusing on a circular database references problem. There will be a whole series of post of the topic on this blog soon.\nBy the way, there were also two other colleagues from Poland. You know Bartosz Ratajczyk [T|B] from Warsaw as he is a dedicated member of \u0026lsquo;Data Community Poland\u0026rsquo; group and he is one of the leaders in Warsaw. He was talking about automating the SSIS deployment process. What I like is how Bartek can explain the topic - he was really well prepared to the session.\nThe last speaker from Poland was Marek Matuszewski [T]. I have met him for the first time in Copenhagen and he was talking about \u0026ldquo;Reaching the Cloud on the BI Rocket\u0026rdquo;. I hope Marek will be able to join the DataCommunity.PL group in Poland and will be a speaker not only at the local meetings but also on the conferences we organize.\nFor the first time in my life, I have won in a raffle. I even took a photo during talking to Kamil \u0026ldquo;I am going to win that one\u0026rdquo;. And I did it!\nThe best part was however after the raffle - tons of hot-dogs and local beers and lot of good discussions with the attendees and other speakers!\nMore details about the sessions:\nKamil Nowinski Bartosz Ratajczyk Marek Matuszewski Damian Widera (U-SQL) Damian Widera (RLS) You can download the slide decks and code and use it.\nBy the way, Kamil is going to fly to #PASS Summit next month and have some great interviews there. Expect great content soon.\nCheers,\nDamian\n","date":"2017-10-12T00:22:07Z","image":"/uploads/2017/10/sqlsat656_print.jpg","permalink":"/2017/10/sqlsaturday-656-denmark-we-were-there/","title":"SQLSaturday #656 Denmark - we were there!"},{"content":"Link to this (past) event: http://www.sqlsaturday.com/642/\nPictures [srizonfbalbum id=17]\n","date":"2017-10-10T20:51:25Z","image":"/uploads/2017/10/sqlsat-642-sofia.jpg","permalink":"/2017/10/sql-saturday-642-sofia-pictures/","title":"SQL Saturday #642 Sofia (pictures)"},{"content":"Link to this (past) event: http://www.sqlsaturday.com/656/\nPictures [srizonfbalbum id=14]\n","date":"2017-10-07T19:31:37Z","image":"/uploads/2017/10/sqlsat-656-denmark.jpg","permalink":"/2017/10/sql-saturday-656-denmark-pictures/","title":"SQL Saturday #656 Denmark (pictures)"},{"content":"On Saturday, October 7th, I’m speaking at SQLSaturday #656 Denmark 2017! I’m presenting Maintenance of a DB project and Continuous Delivery using SSDT.\nI will unveil disadvantages of SSDT and how to work-around them. Also how to maintenance of a database project in the code repository, treat it as a master-version and do deployment evenly and frequently. We will face with the first-world-problem in SSDT, namely: circular dependency - it happens when databases have got references to each other. Further more, I will explain how work with data/dictionary tables - how to manage them, how and when deploy. The best practice from the field.\nI will explain all these topics on the session focused on practical aspects of work with Microsoft Visual Studio Data Tools.\nThat will be double-exciting experience to have speech and speaker dinner on my birthday. Woohoo!\nWill you attend SQLSaturday Denmark? Come and say hello when you see me.\n","date":"2017-10-01T22:08:15Z","image":"/uploads/2017/10/sqlsat656_print.jpg","permalink":"/2017/10/speaking-at-sqlsaturday-656-denmark-2017/","title":"Speaking at SQLSaturday #656 Denmark 2017"},{"content":" Introduction Steph Locke is one of THOSE women in IT.\nShe was awarded an MVP prize for spending plenty of time building communities to provide platforms for people to help each other be better with data.\nAt the beginning of 2017, Steph launched her own consultancy helping people start doing data science.\nShe loves talking about data science, R language, Power BI and many other topics not only as a speaker at conferences.\nThis talk was recorded after SQL Saturday #645 in Manchester, UK, on Saturday 15th July 2017.\nDo you want to know what is the main area of Stephanie\u0026rsquo;s interesting or in what circumstances she begin her own consultancy company? How many books she can read weekly or where has she celebrated her last birthday?\n“Ask SQL Family” podcast discovers all answers for these non-technical questions for you.\nTranscript Kamil Nowinski: So, hi, Stephanie.\nStephanie Locke: Hi!\nKamil Nowinski: Thank you for accepting my invitation.\nStephanie Locke: My pleasure. I like to talk.\nKN: Great! It’s our pleasure basically. We are sitting after the conference in Manchester, Security Saturday. One of the empty rooms. So I hope nobody interrupts us anymore. Let’s start at the beginning and the idea of this interview is to show you to the older people who might not know you very well and to show you not only as a professional but also as a very common person.\nSL: How dare you say I’m common!? [laughs] I’m Welsh, where common is dirt. [laughs] We’re more than common! [laughs]\nKN: Sorry if I use an inappropriate word. [laughs] So tell us your name and where you’re currently living.\nSL: I’m Steph Locke, I live in Cardiff, Wales. I live at home with my\u0026hellip; and I get to work from home most of the time with my husband and two dogs. So I have a great office.\nKN: So how many monitors do you have at home?\nSL: I actually work on just my laptop. Especially because these days, with 4K screens you can fit quite a bit on a 13″ laptop screen. And I work with clients, I work on the road, I work in hotel rooms, so I weaned myself off the dual-monitor or the really big 27″ monitor because it’s just easier if I just make do without it. It’s more convenient. Less stress.\nKN: And what are you doing for a living?\nSL: I started my own date science consultancy in February so since then I’ve been doing a lot of training, helping people upscale date science and associated technologies, helping clients deliver projects, and I’ve been doing a lot of conferences. Which is great because I did that before but now as I’m the boss I get to say “The conference is OK, Steph, you can have some time off after it.” [laughs]\nKN: So when did you start your company?\nSL: February.\nKN: This year?\nSL: Yes. And I haven’t managed to quit yet. So that’s good.\nDamian Widera: We can say that you’re a pretty young boss as February is like 5 months ago.\nSL: And my boss, she’s mean, I only got a Dell Inspiron, I have to earn more money before she allows me to have a much better laptop. [laughs]\nKN: True. So how are things going in your company?\nSL: Good. I was saying earlier that I have enough pipeline until January to cover my salary. So if nothing else, if for some reason nobody else wants to hire me till January, I get to continue doing my own thing for another six months. So that’s fantastic, that’s almost a whole year of being in my own little start-up. So I’m really happy.\nKN: What is that project?\nSL: I’m doing a range of stuff, from training to mentoring, some technical stuff, a whole blend of activities between now and January. Very cool. I like variety.\nKN: What kind of tools do you use during your work? I know you’re involved in data science and you do a lot with R language, Power BI, etc. So what are the tools you most often use?\nSL: R is definitely one of my biggest tools for data science. I do a bit of Python as well. I do a lot of stuff on Azure because quite often with data science, you also have to do a bit of data engineering, so you have to wrangle data into shapes but get it available on a daily, real-time basis. So I do quite a bit of the whole Lambda architecture stuff as well, and write a lot of Azure functions and stream analytics and stuff as well, and Azure ML.\nKN: Basically, the browser. [laughs]\nSL: Yeah, I automate them. Even right down to my presentations, I have a continuous integration and deployment process. So I write stuff for a talk. I do git commit, git push, it goes away to Travis CI, checks that all the code is right and builds and produces my slide deck, and puts it on github pages so that it’s deployed and available. I like automation.\nKN: So you probably use PowerShell?\nSL: Yeah, I do PowerShell too. PowerShell is a great little language. Because I do a lot with Azure, the whole PowerShell modules for that, they change quite a bit and there’s a lot of inconsistencies between them. So it can sometimes be a bit frustrating, writing PowerShell.\nKN: But it’s a very powerful tool, right? Compared to the Command Line.\nDW: Stephanie, do you really like PowerShell or do you must like it because you need it to do your work in Azure? What is your feeling?\nSL: I really like it because I believe in being lazy, so I try to avoid doing repetitive work, or work that is low value. So I like to script away those things and PowerShell helps me do that. So any language which makes it so that I get to do more cool projects is something I’m a fan of.\nDW: Yeah, I think PowerShell is going in a good direction because each UI version is getting better and better. So I can remember PowerShell in version 1 and it was\u0026hellip; yeah it was new, it was fun, but right now you can do almost everything. And the remote stuff is pretty good right now. For example, part of my job is in SharePoint, so configuring SharePoint without PowerShell is impossible so I had to like it, so now I like it.\nSL: Yeah, I think often it takes that “I have to use something” to get you over the reluctance to go through that learning curve. Cause once you’ve gone through that pain of learning the new thing, you can come to appreciate why you learnt it. Whereas before you just see a lot of the learning-curvy things: “But I’m doing fine in my day job. Why would I learn this thing that automates some of my day job? I’m doing it well enough”. Once you’ve gone through it and then you look back, you go: “Wow, did I only use to do that much a day? I’m now doing like five times that much and I get to play with these cool things now.” You think “Yeah, OK, PowerShell has done a great thing for me.” Especially more DBAs should be picking up PowerShell. And more BI people.\nKN: Yeah, especially DBAs. Or accidental DBAs. [laughs] Or even sometimes developer guys because sometimes you can automate deployment processes with PowerShell.\nSL: And it means you don’t have to wait for an infrastructure person to get something done. You can do it yourself.\nDW: Of course if you have permissions or privileges to do so. [laughs] The PowerShell is a must I think. Do you actually like the R language because I see you do a lot of R things also, so do you like it?\nSL: I really like R. It’s a great language. Base R, which is the R that has been around for a very long time, since 2001, was actually written to be backwards compatible with a language called S, and that was written in the late 70s, early 80s, to be interface to a load of Fortran and C algorithms, and to produce charts and stuff. So because this language was written so long ago, there are tons and tons of really weird ways that Base R works. Like to put an object into memory, you make an arrow with a less-than symbol and a hyphen. The reason we do that is because on the keyboards back then, there was one of these symbols when they were typing this S language. So it’s a legacy of something that was around before I was born. So some Base R code can actually support programs that were written before I was born, and they will still yield the same results today. So that’s really amazing but it means the language is crazy. At the base level. So I do a lot with data table and the whole universe of packages called the tidyverse. The tidyverse puts a really sensible veneer of code on top of R. And that makes it\u0026hellip; it’s phenomenal for data manipulation and analysis. Like, I won’t touch SSIS anymore. The GUI is painful. But there are so many\u0026hellip; the static pipeline is\u0026hellip; it’s always been quite a rarity in the work I’ve done, even when I was doing a lot of BI, that R allows me to do dynamic data pipelines and has all sorts of great validation tools. Parallel loads, connectors to basically every single data source one can imagine. So it’s a fantastic tool for ETL as well, and it really makes life so much easier. And it works like PowerShell. It’s a little glue language.\nKN: So far I haven’t seen any solution using R as a pipeline, as a data flow. Have you done something like that?\nSL: Yeah, I’ve built a number of basically like Azure functions. You use that whole micro service approach, so you would just write little R routines for converting one data source to whatever the output is, and then you can do script scheduling or batch processing with them. The nice thing is it’s really easy to build reusable code in R, so we started then having reusable modules which do cleaning in specific ways and routines, so it’s been quite useful for doing in-memory fast ETL.\nKN: So what you basically do is you write an Azure function and then inside the Azure function, you just put the R code and that’s what you do, yes? That’s what I understand.\nSL: No, unfortunately at the moment R is not the supported language in Azure functions. I’ve done a lot of ETL in Azure functions with C# but I use R more in an on-param situation. Or you can push them to Azure batch computes. There is a package that allows you to really easily send jobs to Azure batch. So if you need to do ETL that has to be embarrassingly parallel, big stuff, it’s quite easy to scale out with that. I’m still hoping though that we’ll get R integrated into Azure functions. That would be awesome.\nKN: Exactly, so it looks like a very new feature that could replace the current ETL processes.\nSL: Yeah. The problem I found with SSIS and even Alteryx and stuff is, it kind of condenses down who can do ETL. But quite often devs need to do some ETL as well, data scientists need to do some ETL, your BAs can do some. And if you can make it so that you’re processing engine will allow any number of languages, then your devs can write a bit in C#, your data scientists can write a bit in R or Python, your BAs can use like logic apps or something. It’s chaos if you do it wrong. But that kind of more distributed ETL fabric I think just makes it so much easier for people to flex and grow what they’re doing.\nDW: Yes, we have different people with different technical knowledge, using different tools. So the ETL processes can be very complicated because they can use their own favourite languages to achieve the same goal. We just want to gather and transform some data and that’s what we like to do, so yeah.\nKN: During the latest interview, you have said that the language is the most interesting innovate area where the data science has been applied to. Could you explain this thought?\nSL: So, language is such an important component of our lives. Most people think in\u0026hellip; actually think in words, apparently very few people think mostly in images. So all our mental constructs is language. When we communicate ideas, whether it’s written or speech, or even body language, these are all things that we are doing to communicate. And it’s such an incredibly complex, diverse area. Like if I say: “That’s great.” or “That’s great!” or “That’s great\u0026hellip;”\nKN: Different expressions.\nSL: Yeah. So being able to chuck computers at the problem of language, being able to real-time translate, and then being able to real-time translate the nuance then as well, so the tone\u0026hellip; because how people are sarcastic in English is different to how people are sarcastic in Polish. So being able to keep up with all the different ways people speak, deciphering accents, real-time translation, and then real-time translation of emotions and stuff\u0026hellip;\nKN: I would imagine that would be the next step. That’s why we sometimes don’t understand English jokes.\nSL: [laughs] I spend a lot of my time building linear or logistic regression models to predict things and it’s great. It helps banks make more money, helps people make customers more happy. The Microsoft thing of Internet of cows, where they stuck thick bits on cows and analysed how they moved so they could work out when to send an artificial bull around, improve milk yields. We’re doing all these pretty\u0026hellip; some things are a bit dry, some things are very cool, but I think the utopia of everybody being able to understand everybody else, the Tower of Babel situation or the Douglas Adams Babel Fish, that is just the coolest area that we can be applying machine learning to. I think it’s a really laudable goal.\nKN: I heard that bots are your area of interest.\nSL: Bots! Yeah, I’ve become really interested in bots quite recently because I don’t like Siri and Cortana and stuff. So I can read and type much faster than I can talk, and I can talk pretty quickly. So I wanna be able to get my job done as quickly as I can, so I ready a lot, and I use a lot of Linux, so I’m used to my Command Line and PowerShell and everything. So chatbots are basically text command lines.\nKN: It’s a natural thing, a first idea.\nSL: Yeah, it’s a way of me interfacing with systems in a text way. It’s much quicker for me than going and phoning somebody if a chatbot can answer my question. So it’s an area I’m starting to build stuff in, so I started small, I built an FAQ bot, and my aim is to build to a bot that will allow you to say like “Show me sales by such and such,” so like the natural language Q\u0026amp;A stuff in Power BI, and charts returned to you and stuff. So anybody can talk to the company chatbot with maybe like a tabular module or something behind it with measures and things and be able to get analysis back. That’s my aim, that’s what I’m working towards, and I’m gonna be doing a load of learning next month towards upping my C# or my Node.js skills to be able to get closer to that.\nKN: That’s an interesting idea. Do you know how many successful implementations are there across the world of this kind of bots?\nSL: I have no clue. I imagine it’s probably doubling each year right now, if nothing else. The growth rate on it is really huge and Microsoft are actually making it really easy so their bot framework sits on Azure functions, which is partly why I like it so much because I really like Azure functions. So just integrates really quite well with stuff that I already started learning about last year, further sort of distributed ETL. So yeah, it’s a really big growth area.\nKN: When did you start focusing on data science itself and on that area? Or maybe it was something earlier?\nSL: I started about 4 years ago now. So before then, I’d done some forecasting, some RepEntity models and things. But it was always “I think we need this, so I’m gonna go ahead and build it in my spare time” kind of thing, for work. Then I got to do something a bit more serious, so I built predictive models to say who was not going to pay their loan. When were they likely to stop paying their loan. And as a result, I was able to put these together, so that we could go to the regulator to say: “Here is our rating system, and here is what we think we need to hold against the downturn, so how much money do we need to keep behind.” And that saved the company something like 40 million pounds. There were able to take that just sitting around and put it to use. And then I went to a second mortgage start-up, so I built a real-time rate-risk pricing model. So what that meant was as every application came in for a loan, everybody got a price which was appropriate for their risk balance for company’s requirements, and also took into account competitors. Real-time. That was built in R with a MariaDB back-end. [laughs]\nKN: And in this place we’ll do like “beep-beep.” [laughs]\nSL: Yeah, I would understand. Yeah, I made an API in R with Apache because the Azure ML wasn’t out or anything by then, and it was all on-param and had less than like a 200 ms response rate and that still did a whole load of reads and writes to the database as well. So I was pretty proud of that. I did that in less than 6 months. I was like “Yeah!”. The whole application process took something like 30 seconds so was just like “Yeah, I can do whatever I want, I can do crazy amounts of extra stuff, up to a second and nobody will notice.” Then I went to work for Mango Solutions, so I was a principal consultant for them and I did data science consultancy for them for a while. Then I went to a security start-up and helped build a real-time anomaly detection system, as people were browsing, who is going to file sharing sites in an unusual amount, or going to a new file sharing site. Or like who is\u0026hellip; who might be sending data out of company resources. Or who has started using their computers and like, are we picking up lots more traffic to malware sites. Or worked with education customers in the UK and now they have a legal obligation to detect radicalisation, so who is going to radical sites in a way that doesn’t look just like accidentally hitting it, and firing a notification to the admins or putting an audit log in place to say this is happening. So I did a load of that, and I was looking and I was thinking: “Do I apply for Microsoft, do I do my own thing?” Microsoft never got back to me [laughs] so I thought “OK, I’ll do my own thing.” [laughs]\nKN: You started talking about the last company.\nSL: I actually applied to Microsoft at the same time when I was thinking about what I wanted to do, since I was coming up 30 and my decade plans for my 30s was to\u0026hellip;\nAnd Microsoft didn’t get back to you, but I’m not quite sure about this because you have been awarded the MVP, so basically Microsoft did get back to you.\nDW: On a different level.\nSL: Yeah, it’s actually the closest I’ve come with my MVP status of thinking: “Don’t you know who I am?” I was like, when I applied to Microsoft and they ignored me and my application, I didn’t even get a “Thanks, but no, thanks.” [laughs]\nKN: So you just applied to Microsoft with the MVP award?\nSL: Yeah! I had the MVP and I thought this lead data scientist role in their team next to the Evangelist team will be like a nice fit for me. But they never got back to me, I was like “OK.” Well, it’s worked out quite well for me. It made my choices easier. So then I launched my business and yeah. I’m quite happy with the way it worked.\nKN: OK, and talking about spare time, because we mentioned it before, what about your work-life balance? Could be a little bit difficult now for you.\nSL: Yeah, I tend to go through\u0026hellip; I’m very overall balanced but not day-to-day balanced, so one day I might be on site or at a conference, then I might be home for a few days. So I’m looking forward to spending August at home, not going on site to any clients, building lots of my training materials, developing lots of stuff for kind of a Q4 programme.\nKN: I saw that because you shared your calendar on the website, so August is empty.\nSL: Yes, and given that September has two online conferences, potentially four in-person conferences\u0026hellip;\nKN: Including Katowice in Poland.\nSL: Including Poland. And I’m looking forward to that. That’s my first time going to Poland. Yeah, so September’s really busy. So because of that I made August very quiet. So overall it’s like doing two conferences a month. I just happen to be doing four in one run. [laughs]\nKN: You can spend more time on your hobby. What is your hobby?\nSL: I read a lot, and I actually found an author recently who can cope with my reading speed, so there is an author called Michael Anderle and he writes these really fun books about vampires and werewolves being a product of alien intervention, so it’s a sci-fi with vampires in. And since he started writing the books in 2015, there are 17 books in the main series that he’s written and he has co-written with something like ten other authors another 30 books. There are now something like eight series in the universe. I think it’s called The Kurtherian Universe. So around these books. And now they’re kind of outputting one book a week, which is like a couple of hours’ reading for me, but he’s the only author who comes close.\nKN: Crazy speed of writing. [laughs]\nSL: Crazy speed of writing, which means I can do my crazy speed of reading.\nKN: Maybe he has a bot to write his books.\nSL: Maybe, yeah. Now he has all these authors as well working on it, so it’s really awesome. I play a lot of board games.\nKN: What is your favourite?\nSL: That’s a tough one. I really like my abstract games, but probably my most favourite game is Cribbage, which is a classic card game, and Star Realms, which is kind of a deck builder, so it’s very much like a resource optimisation challenge and everything. It puts my data science brain to work in games. There are some games that I don’t play very often because I’m too good at erm\u0026hellip; There’s a game called Camel Up, where you basically place bets on how soon these camels will finish doing a race. But because the amount of moves everything can take is really constrained, because I can do all the probabilities and the conditional probabilities quite quickly in my head, I’m too good at the game and I don’t like games which I win at all the time. Something that you always win at isn’t worth doing.\nKN: Basically, there are no people who want to play with you. [laughs]\nSL: No! My husband would like to play the game but I’m like “Dude, I win. If I always win, that means you always lose. Why would you always want to lose?!” That’s worse than always wanting to win. I want a challenge.\nDW: So the challenge could be if he took you to a casino next time you are in Poland and you could do the probability things, the data science in your head and maybe we could even win something. But I think it’s too risky.\nSL: Yeah, I don’t do betting generally, just because it’s always weighted against the punter. You actually make more money just by investing in an automated tracker funds. Stock brokers, they usually, like hedge fund managers and fund managers, they get short-term gains but overall they don’t perform that much better than the stock market, whereas the automated trackers, the ones that computers pick the stocks and everything, often do a lot better. So just invest your money in an automated tracker fund like Vanguard. That’s the best way to make money, instead of betting in casinos.\nDW: Yeah, betting in casinos, you never win in the long-term. You can win one or two games or two bets but at the end - you’ll always be a loser.\nKN: OK, but one win is enough for me. [laughs]\nDW: One big win is enough.\nSL: But most people can’t stop at one win. It’s one of the biases we have: “Oh, I’ve won one game out of three, I must be able to win two games out of four or two games out of five” and then it just carries on going and every win reaffirms the value of gambling, so people just carry on, even though\u0026hellip; they go “You’ve lost seven times. Why do you still think this is a good idea? You have less money than you started off with.” So I don’t do gambling very often.\nKN: Very good. One needs to be very careful with that. It’s very addictive, I heard. I only heard that, never tried.\nSL: Wanna bet?\nKN: No! Maybe next time. Maybe when we stop recording. [laughs]\nKN: When did you start with SQL Server?\nSL: I was 20? I was doing some analytical work at confused.com, so I was doing a lot of work. I was doing web-scraping in Excel for one part of my day and then writing emails and PR stories in another part of the day. And I wanted to get more and more data to go with my spreadsheets so I got somebody to teach me some SQL and I got access to the database, and just became the horrible person who knows enough to be dangerous and leaves queries running. And then I moved in to the BI team after that to get more data, to be able to do more things. And yeah, I just keep moving sideways, I’m kind of into things that are interesting and challenging.\nKN: And what hints would you give to young people who want to start on that market, using SQL Server, or using that kind of stuff?\nSL: Get DreamSpark.\nKN: What? [laughs]\nSL: So, Microsoft have these packs of credits that students can get called DreamSpark, so they can get like Azure credits, as much as the MSDN thing, for free. So if you wanna learn, go and do.\nKN: But it’s only for one month?\nSL: No! Basically for the duration you are a student, you can get the DreamSpark thing. So if you’re a student, go forth and do. Projects that you can show people that you spent the time building things is so much better than going “Oh, I did a module on data basis” because usually the person who taught you, hasn’t actually written a real query in about 30 years. I’m very cynical of mainly computer science degrees and the quality of the IT person that comes out. I find that generally they believe they’re really good at computers, but having no commercial or practical experience quite often. They need breaking the belief they know a lot, especially in databases.\nKN: Are you a perfectionist?\nSL: Relatively so. I try to set myself limits and deadlines so that if I’m looking like I’m not gonna meet this deadline, then I have to stop trying to get it awesome and just try and get something that works. Cause I have a tendency to want to make brilliant things when good enough is usually the right answer. So I try and set myself stop points. Coping strategies of being a perfectionist. [laughs]\nKN: Any strategy that gives you your goals is a good strategy.\nDW: So maybe I can ask something. Because I’m really curious about how you started being a speaker. When was your first conference? Were you scared about talking to the people when the room is crowded? What was your feeling then?\nSL: So my first time speaking properly was at the Cardiff User Group, back before I ran it. I was a bit nervous so what I did was I bought a bag of chocolates along and then I said “Everybody who asks a question gets a chocolate” so they weren’t listening to me, they were trying to think of questions to ask. And that just made it so much better. [laughs]\nKN: And so much more fun probably.\nYeah. So a data-addicted woman and a chocolate-addicted woman.\nSL: Yes. I’m a feeder. I believe “well-fed, full people – happy people.” They can’t get up to mischief and they don’t complain. So when I run conferences, I always try to make sure that lunch and everything is really good because that will make up for so much stuff. If the food’s bad then even the best speakers\u0026hellip; their feedback gets lower. Because people look grumpy without good food.\nKN: Exactly. We know that and we have that feedback from the SQLDay conference. Is that right, Damian?\nDW: Yes, most of the statements are about the food or coffee, or drinks. Because we know that the content, the sessions are OK, so all the questions are mostly about “Make better food next time,” something like this. And you know, if you have 800 people than it’s a logistics problem how to make everybody happy, so the lunch is not so long, it’s one hour, so how are we going to do this, feed everybody? Maybe one more question, just about the conference. Do you like talking to people, do you like talking to people on conferences? And what do you think about networking? How important is this for you?\nSL: In some ways I’m very anti-social. I love being at home, warm with a book and a nice coffee or a nice drink. But I also like being out and meeting people. For my 30th birthday, I went to a SQL conference, so SQLGrillen was William Durkin [B|T] basically throwing me a barbecue and inviting lots of my friends. So we spent a day learning, I presented on my birthday, we had drinks before that, drinks after that, then it was a medieval festival, so we spent all day enjoying ourselves.\nKN: So you were celebrating your 18th birthday with the SQL family.\nSL: I see and consider a lot of the SQL family my friends more than I have just like non-SQL, normal friends. So it was a great way to spend my birthday, in the company of people who are my friends. And on the networking thing, I still find it difficult to just approach people, to talk to them, but that’s where being a speaker really comes in handy. Because if you’re nervous about getting up in front and speaking in front of people, once you’ve done that, it gives everybody else a reason to come and talk to you. So then you never have to do actually the much more scary thing of going up to a group of people and going “Hi, I’m Steph! What do you do?” So yeah, speaking or volunteering, I really recommend volunteering an event, putting on a helper shirt really helps you network people and overcomes a lot of anxieties about it.\nKN: Thank you very much, Stephanie. It was our pleasure.\nDW: Thank you.\nKN: At the end of our conversation tell us where we can find you and where people can find you.\nSL: Sure. You got my blog, itsalocke.com, and at the moment I also have a site full of my presentations on lockedata.uk, and you can follow me on Twitter at SteffLocke or at LockeData.\nKN: OK, thank you very much indeed.\nSL: My pleasure. Thank you for having me.\nDW: Thank you very much and see you soon!\nSL: Yeah, bye! See you soon!\nUseful links: Steph Locke’s websites: Company | Locke Data Talks\nSteph Locke’s Twitters: Private | CompanySteph Locke’s GIT repository: GitHub account\n","date":"2017-09-16T11:45:19Z","image":"/uploads/2017/09/IMG_5984.jpg","permalink":"/2017/09/asf-003-stephanie-locke-interview/","title":"ASF 003: Stephanie Locke interview"},{"content":"Today, I\u0026rsquo;d like to describe briefly one thing with SQLPackage.exe.\nI know\u0026hellip; It\u0026rsquo;s not precisely related to PowerShell (this week topic), but\u0026hellip; you can use PowerShell to run it (and honestly I\u0026rsquo;m doing it via Octopus).\nThis application can be used to deploy (publish) Microsoft SQL database saved in SSDT.\nIn many ways. No matter what type of script you\u0026rsquo;re using: batch (old-fashioned, I know), PowerShell (cool!) or you\u0026rsquo;ll run this exe from other custom app written in C#, for example. All the ways leading to the goal are fine.\nA compiled version of such database dwells as a DACPAC file form, which contains all objects of database required to its reflect into the target.\nWhat\u0026rsquo;s important in this post\u0026rsquo;s context - the file can contain relations to the other databases:\n1 2 3 4 5 6 CREATE PROCEDURE [dbo].[getCustomerInfo] @customerId int AS SELECT [CustomerId], [CountryCode], [FirstName], [Surname], [isActive], [CustomerTypeCode], [Twitter] FROM [$(CRM)].dbo.Customer; RETURN 0 Action: Report, script, publish There are more than 3 actions possible with SQLPackage.exe, but from a deployment perspective, we are interested in:\n• DeployReport - creates an XML report file with describes changes that would be done during publishing\n• Script - creates an incremental update T-SQL script that updates the target schema of a target database\n• Publish - incrementally updates a database schema to match the schema of a source .dacpac file.\nSince the update of October 2016 - all above actions can be done in one pass. Believe me, it saves a lot of time.\nPublish So, how to publish database? You can use Visual Studio, right-click on selected database and choose to Publish:\nThen you can click \u0026ldquo;Generate Script\u0026rdquo; or \u0026ldquo;Publish\u0026rdquo;. Everything gonna be fine.\nBut hey\u0026hellip; We love automation, right? Let\u0026rsquo;s have a look on that batch script (@Rob [B|T], I promise to deliver PowerShell version soon ;) ):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 rem DeployDACPAC.bat echo off set prg=\u0026#34;c:\\Program Files (x86)\\Microsoft Visual Studio 12.0\\Common7\\IDE\\Extensions\\Microsoft\\SQLDB\\DAC\\130\\sqlpackage.exe\u0026#34; set dir=Q:\\Dropbox\\Post\\ set output=Q:\\Dropbox\\Post\\output\\ SET prgtmp=%prg:\\=\\\\% echo This DacFX will be used: %prg% WMIC DATAFILE WHERE name=%prgtmp% get Version /format:Textvaluelist set db=ContosoRetailDW set dacpac=\u0026#34;%dir%%db%\\bin\\Debug\\%db%.dacpac\u0026#34; set profile=\u0026#34;%dir%%db%\\%db%.publish.xml\u0026#34; echo === Database: %db% === echo Start: %date% %time% echo Profile: %profile% %prg% /Action:Publish /SourceFile:%dacpac% /Profile:%profile% /dsp:\u0026#34;%output%%db%.publish.sql\u0026#34; /drp:\u0026#34;%output%%db%.report.xml\u0026#34; Wait, wait\u0026hellip; What\u0026rsquo;s that error?\n*** Error parsing connection string: An item with the same key has already be added.. As you can see on the picture below - the project contains two variables possessing the same value. It can be a value for some field, part of stored procedure or function, or (more frequently used) - database name referral.\nThis is a very simple example, but in the real world scenario - you can have tons of such variables. Those variables can be scattered after many database projects related to each other. Hence, the variables can come from many (related) projects.\nWhat goes wrong? The publication from UI (Visual Studio) would be gone properly with exactly the same projects, with exactly the same publish profile.\nWhat we\u0026rsquo;ve done wrong? Where is the bug?\nApparently, SQLPackage.exe is case-sensitive, whereas Visual Studio does not.\nHow to fix it? You must normalize that variables in all your database projects. I\u0026rsquo;d suggest using upper-case always:\nUnfortunately, sometimes it\u0026rsquo;s not possible to change it from UI (for some enigmatic reasons), then you can edit (be careful) project file using Notepad++ or Visual Studio Code. It would be enough to change attribute \u0026ldquo;Include\u0026rdquo; in \u0026ldquo;SqlCmdVariable\u0026rdquo; element:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 \u0026lt;ItemGroup\u0026gt; \u0026lt;ProjectReference Include=\u0026#34;..\\CRM\\CRM.sqlproj\u0026#34;\u0026gt; \u0026lt;Name\u0026gt;RM\u0026lt;/Name\u0026gt; \u0026lt;Project\u0026gt;{891a1f33-a9b8-4a4d-b2b8-e45c917a8b0d}\u0026lt;/Project\u0026gt; \u0026lt;Private\u0026gt;True\u0026lt;/Private\u0026gt; \u0026lt;SuppressMissingDependenciesErrors\u0026gt;False\u0026lt;/SuppressMissingDependenciesErrors\u0026gt; \u0026lt;DatabaseSqlCmdVariable\u0026gt;Crm\u0026lt;/DatabaseSqlCmdVariable\u0026gt; \u0026lt;/ProjectReference\u0026gt; \u0026lt;/ItemGroup\u0026gt; \u0026lt;ItemGroup\u0026gt; \u0026lt;SqlCmdVariable Include=\u0026#34;Crm\u0026#34;\u0026gt;; \u0026lt;DefaultValue\u0026gt;CRM\u0026lt;/DefaultValue\u0026gt; \u0026lt;Value\u0026gt;$(SqlCmdVar__1)\u0026lt;/Value\u0026gt; \u0026lt;/SqlCmdVariable\u0026gt; \u0026lt;/ItemGroup\u0026gt; \u0026lt;/Project\u0026gt; Then you need to remove the duplicated key from the publish file:\nWhat next? Do you need more? No problem at all.\nI\u0026rsquo;ll prepare the whole series about SSDT and Continuous Delivery for databases soon. Stay tuned!\nUpdated @ 08/11/2019\nThis post is already part of the whole series about SSDT.\n","date":"2017-09-12T09:18:01Z","image":"/uploads/2017/09/tt_300_11FF5A1D.jpg","permalink":"/2017/09/tsql2sday-how-sqlpackage-exe-can-spoil-your-deployment/","title":"#TSQL2sDay - How SQLPackage.exe can spoil your deployment"},{"content":"Greg Low from SQL Down Under released a new version of his tool called \u0026ldquo;SDU Tools\u0026rdquo;.\nThe tool contains many interesting and useful functions and stored procedures you can use in many projects.\nFor instance:\n1 2 3 4 5 SELECT SDU_Tools.DigitsOnly(\u0026#39;Hello20834There 234\u0026#39;, 1); SELECT SDU_Tools.DigitsOnly(\u0026#39;(425) 902-2322\u0026#39;, 1); SELECT SDU_Tools.DigitsOnly(\u0026#39;+1 (425) 902-2322\u0026#39;, 1); SELECT SDU_Tools.DigitsOnly(\u0026#39;+1 (425) 902-2322\u0026#39;, 0); GO The code is fully documented itself and if you need more support or usage examples - you can find here.\nCSV reader What I was interested the most - it\u0026rsquo;s a new feature in the latest version 5.0 (August 2017).\nThe new function is ReadCSVFile.\nI\u0026rsquo;ve decided to give it a closer look and test as it would be very useful for at least two reasons:\nEntire tool doesn\u0026rsquo;t need CLR enabled All you need to have is a SQL engine, without SSIS Let\u0026rsquo;s get the party started At the beginning, let\u0026rsquo;s do some simple test and prepare the file for that.\nI\u0026rsquo;ve downloaded a CSV file that contains list of presidents of The USA.\n[USPresident-Wikipedia-URLs-Thmbs-HS]\n1 2 EXEC SDU_Tools.ReadCSVFile @Filepath = N\u0026#39;x:\\SDU_Tools\\USPresident-Wikipedia.csv\u0026#39;, @Delimiter = \u0026#39;,\u0026#39;, @TrimOutput = 1, @IsFileUnicode = 0, @RowsToSkip = 0; That is cool! Look\u0026rsquo;s almost perfect.\nI would like to change the column names from ColumnXX to appropriate name from the file. Can I? There is a parameter called @RowsToSkip - let\u0026rsquo;s check it.\n1 2 EXEC SDU_Tools.ReadCSVFile @Filepath = N\u0026#39;x:\\SDU_Tools\\USPresident-Wikipedia.csv\u0026#39;, @Delimiter = \u0026#39;,\u0026#39;, @TrimOutput = 1, @IsFileUnicode = 0, @RowsToSkip = 1; Closer. It skipped my header row, but column names remain as previously. Hence, there is no that option with this Stored Procedure to achieve that, but we apply one trick.\nRename columns You can create (temporary) table with all 50 columns, exec procedure loading the outcome over there. But this doesn\u0026rsquo;t look funny:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 EXEC SDU_Tools.DropTemporaryTableIfExists N\u0026#39;#tmp\u0026#39;; CREATE TABLE #tmp ( [Column01] VARCHAR(250), [Column02] VARCHAR(250), [Column03] VARCHAR(250), [Column04] VARCHAR(250), [Column05] VARCHAR(250), [Column06] VARCHAR(250), [Column07] VARCHAR(250), [Column08] VARCHAR(250), [Column09] VARCHAR(250), [Column10] VARCHAR(250), [Column11] VARCHAR(250), [Column12] VARCHAR(250), [Column13] VARCHAR(250), [Column14] VARCHAR(250), [Column15] VARCHAR(250), [Column16] VARCHAR(250), [Column17] VARCHAR(250), [Column18] VARCHAR(250), [Column19] VARCHAR(250), [Column20] VARCHAR(250), [Column21] VARCHAR(250), [Column22] VARCHAR(250), [Column23] VARCHAR(250), [Column24] VARCHAR(250), [Column25] VARCHAR(250), [Column26] VARCHAR(250), [Column27] VARCHAR(250), [Column28] VARCHAR(250), [Column29] VARCHAR(250), [Column30] VARCHAR(250), [Column31] VARCHAR(250), [Column32] VARCHAR(250), [Column33] VARCHAR(250), [Column34] VARCHAR(250), [Column35] VARCHAR(250), [Column36] VARCHAR(250), [Column37] VARCHAR(250), [Column38] VARCHAR(250), [Column39] VARCHAR(250), [Column40] VARCHAR(250), [Column41] VARCHAR(250), [Column42] VARCHAR(250), [Column43] VARCHAR(250), [Column44] VARCHAR(250), [Column45] VARCHAR(250), [Column46] VARCHAR(250), [Column47] VARCHAR(250), [Column48] VARCHAR(250), [Column49] VARCHAR(250), [Column50] VARCHAR(250) ); INSERT INTO #tmp EXEC SDU_Tools.ReadCSVFile @Filepath = N\u0026#39;x:\\SDU_Tools\\USPresident-Wikipedia.csv\u0026#39;, @Delimiter = \u0026#39;,\u0026#39;, @TrimOutput = 1, @IsFileUnicode = 0, @RowsToSkip = 1; CREATE TABLE dbo.SomeTarget ( [Presidency]\tVARCHAR(250) NULL, [President]\tVARCHAR(250) NULL, [Wikipedia Entry]\tVARCHAR(250) NULL, [Took office]\tDATETIME NULL, [Left office]\tDATETIME NULL, [Party]\tVARCHAR(250) NULL, [Portrait]\tVARCHAR(250) NULL, [Thumbnail]\tVARCHAR(250) NULL, [Home State]\tVARCHAR(250) NULL ); SET DATEFORMAT DMY; INSERT INTO dbo.SomeTarget ( [Presidency], [President]\t, [Wikipedia Entry], [Took office], [Left office], [Party], [Portrait], [Thumbnail], [Home State] ) SELECT [Presidency]\t= [Column01], [President]\t= [Column02], [Wikipedia Entry]\t= [Column03], [Took office]\t= TRY_CAST([Column04] AS DATE), [Left office]\t= TRY_CAST([Column05] AS DATE), [Party]\t= [Column06], [Portrait]\t= [Column07], [Thumbnail]\t= [Column08], [Home State]\t= [Column09] FROM #tmp; Performance From the beginning when Greg mentioned about this function (before its official release), I was very curious whether I\u0026rsquo;d be able to load bigger files, let\u0026rsquo;s say 500MB?\nTo be honest, I don\u0026rsquo;t believe in that as SQL engine would have to load entire file into memory first.\nHow long it takes now for those 44 rows?\n1 2 3 SET STATISTICS IO ON; SET STATISTICS TIME ON; GO Result:\n1 2 3 4 5 SQL Server Execution Times: CPU time = 47 ms, elapsed time = 154 ms. (44 row(s) affected) I gonna multiplicate rows in the file and make 4400 of them.\n1 2 3 4 5 6 7 --4400x EXEC SDU_Tools.ReadCSVFile @Filepath = N\u0026#39;x:\\USPresident-Wikipedia 4400.csv\u0026#39;, @Delimiter = \u0026#39;,\u0026#39;, @TrimOutput = 1, @IsFileUnicode = 0, @RowsToSkip = 1; SQL Server Execution Times: CPU time = 10452 ms, elapsed time = 10626 ms. We can already see that this solution won\u0026rsquo;t replace our good old SSIS.\nAnyway, let\u0026rsquo;s do the last test with even more amount of rows to have a comparison.\n1 2 3 4 5 6 --22220 rows SET STATISTICS IO OFF; SET STATISTICS TIME OFF; EXEC SDU_Tools.ReadCSVFile @Filepath = N\u0026#39;x:\\USPresident-Wikipedia 22220.csv\u0026#39;, @Delimiter = \u0026#39;,\u0026#39;, @TrimOutput = 1, @IsFileUnicode = 0, @RowsToSkip = 1; On my laptop that tests took almost 3 minutes (2:52). Bad result.\nI guessing that it wasn\u0026rsquo;t an intention of the author to provide smart replacement for SSIS for reading CSV files. However, you can use this method for dev purposes and to get a quick preview of CSV file.\nAdditional quotes and NULL CSV is a Comma-Separated Values, right?\nHence you need to have a separator for rows and columns. Both different.\nWhat if your data contains char using as a separator? In that scenarios the value should be encapsulated by additional characters:\nOtherwise, the reader will interpret that value incorrectly (row id 5).\nI only wanted to check if the tool which I\u0026rsquo;m testing today is resistant against that.\n1 2 EXEC SDU_Tools.ReadCSVFile @Filepath = N\u0026#39;x:\\USPresident-Quotes.csv\u0026#39;, @Delimiter = \u0026#39;,\u0026#39;, @TrimOutput = 1, @IsFileUnicode = 0, @RowsToSkip = 1; This is a very common scenario when people trying to understand what has happened and why the ETL process doesn\u0026rsquo;t load data correctly and/or why failing when tries to convert string-value to date or numeric value.\nAs you can see above - the procedure can\u0026rsquo;t cope such cases.\nBut, to be fair - I\u0026rsquo;ve never expected such complex processing in the function who do not use CLR/.NET libraries but purely T-SQL.\nSummary This function has certain limitation. But hey. It\u0026rsquo;s only T-SQL, there is no .NET code in there.\nIt can help you review or/and load some CSV files, but not very big and non-complex one.\nHopefully, the next release of this method will fix \u0026ldquo;quote-issue\u0026rdquo; mentioned above.\nResources USPresident-Wikipedia - ZIP file (contains CSV files)\nSDU Tools.\n","date":"2017-09-04T05:00:57Z","image":"/uploads/2017/09/spices-1914130_1920.jpg","permalink":"/2017/09/csv-reader-from-sdu-tools/","title":"CSV reader from SDU Tools"},{"content":"Certainly, you have heard about TRUNCATE. On a table.\nAlso probably you\u0026rsquo;ve already heard about TRUNCATE on partitions. This feature is in the latest version (latest, I mean 2016 as the 2017 has not been released yet) of SQL Server. And it\u0026rsquo;s very cool.\nBut this post is not about this statement. I only mention about it to give you an opportunity to compare both solution: the newest-one-cool-statement TRUNCATE PARTITION and my-own-written-old-targetted version.\nLet’s take a look how does the syntax looks like in SQL Server 2016 and later:\n1 2 3 4 5 6 7 8 9 10 11 -- Syntax for SQL Server and Azure SQL Database TRUNCATE TABLE [ { database_name .[ schema_name ] . | schema_name . } ] table_name [ WITH ( PARTITIONS ( { \u0026lt;partition_number_expression\u0026gt; | \u0026lt;range\u0026gt; } [ , ...n ] ) ) ] [ ; ] \u0026lt;range\u0026gt; ::= \u0026lt;partition_number_expression\u0026gt; TO \u0026lt;partition_number_expression\u0026gt; So, let prepare one example with data:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 CREATE PARTITION FUNCTION pfQuarter(INT) AS RANGE RIGHT FOR VALUES (201701, 201702, 201703, 201704) GO CREATE PARTITION SCHEME psByQuarter AS PARTITION pfQuarter ALL TO ( [PRIMARY] ) GO CREATE TABLE [dbo].[BigOrderTable]( [ID] BIGINT NOT NULL, [OrderNumber] VARCHAR(30) NOT NULL, [OrderDate] DATETIME NOT NULL, [QuarterNum] AS (datepart(year,[OrderDate])*(100)+datepart(quarter,[OrderDate])) PERSISTED, CONSTRAINT [PK_dbo_BigOrderTable] PRIMARY KEY CLUSTERED ([ID] ASC, [QuarterNum] ASC), ) ON psByQuarter ([QuarterNum]) GO --Let’s fill the [BigOrderTable] table with a few rows: INSERT [dbo].[BigOrderTable] ([ID], [OrderNumber], [OrderDate], [Notes]) SELECT message_id * 10000 + language_id as [ID] , \u0026#39;ORD/\u0026#39; + CAST(message_id AS varchar(20)) + \u0026#39;/\u0026#39; + CAST(language_id AS varchar(20)) as [OrderNumber] , DATEADD(day, ABS(CHECKSUM(NEWID())) % 365, \u0026#39;20170101\u0026#39;) as [OrderDate] , LEFT([text], 500) as [Notes] FROM sys.messages; Hence, a simple example would look like below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 --Show the entire table SELECT * FROM [dbo].[BigOrderTable] GO --Show me how many rows are per partition SELECT partition_number, rows FROM sys.partitions WHERE OBJECT_NAME(OBJECT_ID)=\u0026#39;BigOrderTable\u0026#39;; GO --You can truncate one partition... TRUNCATE TABLE [dbo].[BigOrderTable] WITH (PARTITIONS (2)); GO --...or range of partitions TRUNCATE TABLE [dbo].[BigOrderTable] WITH (PARTITIONS (2 TO 3)); GO Table is nothing but a set of one partition, so you can use above operation for tables as well.\nFine. But this feature is available in 2016. And now is the year 2017. It\u0026rsquo;s a quite new feature, and believe me - some companies still use 2005 (yes, that not supported one) or even 2000. Should say at the moment: god, bless America. Eeeee\u0026hellip; no. Better: god, bless DBA\u0026rsquo;s who have to use those versions.\nSo, what when we need or want to use such trick like truncate but for selected partition?\nCASE: Truncate table with SQL Server before 2016 Let\u0026rsquo;s do some simple (but not trivial) trick.\nDo create the same table as our target as a distinct object. I suggest using separate schema with the same table name:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 --Create table in \u0026#39;part\u0026#39; schema CREATE SCHEMA [part] GO --Create clone of the table CREATE TABLE [part].[BigOrderTable] ( [ID] BIGINT NOT NULL, [OrderNumber] VARCHAR(30) NOT NULL, [OrderDate] DATETIME NOT NULL, [QuarterNum] AS (DATEPART(YEAR,[OrderDate])*(100)+DATEPART(QUARTER,[OrderDate])) PERSISTED, [Notes] CHAR(500) NULL, CONSTRAINT [PK_dbo_BigOrderTable] PRIMARY KEY CLUSTERED ([ID] ASC, [QuarterNum] ASC), ) GO Fine. But, \u0026ldquo;how can I truncate partition using SQL Server older than 2016 version?\u0026rdquo; - you would ask. I\u0026rsquo;ve just revealed that secret in the text above partly and here is a low-budget counterpart of TRUNCATE TABLE WITH PARTITION for servers \u0026lt;2016:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 USE DEMO GO ------------------------------------------------ -- PROCEDURE: [TruncatePart_BigOrderTable] ------------------------------------------------ ALTER PROCEDURE [dbo].[TruncatePart_BigOrderTable] @Quarter INT AS --Drop check constraint IF EXISTS (SELECT [name] FROM sys.objects WHERE [name] = \u0026#39;CK_BigOrderTable_Quarter\u0026#39; and [type]=\u0026#39;C\u0026#39;) ALTER TABLE [part].[BigOrderTable] DROP CONSTRAINT CK_BigOrderTable_Quarter --Truncate partition TRUNCATE TABLE [part].[BigOrderTable]; --Switch off the partition from the table DECLARE @PartNo INT; SELECT @PartNo = $PARTITION.pfQuarter (@Quarter); ALTER TABLE [dbo].[BigOrderTable] SWITCH PARTITION @PartNo TO [part].[BigOrderTable]; --Truncate partition TRUNCATE TABLE [part].[BigOrderTable]; --Add CHECK constraint again DECLARE @ConstraintSQL NVARCHAR(500); SET @ConstraintSQL = N\u0026#39;ALTER TABLE [part].[BigOrderTable] ADD CONSTRAINT CK_BigOrderTable_Quarter CHECK ([QuarterNum] = \u0026#39; + CAST(@Quarter AS VARCHAR(20)) + \u0026#39;);\u0026#39; EXECUTE sp_executesql @ConstraintSQL; --Switch on the partition into the table ALTER TABLE [part].[BigOrderTable] SWITCH TO [dbo].[BigOrderTable] PARTITION @PartNo; PRINT \u0026#39;Partition (\u0026#39; + CAST(@PartNo AS VARCHAR(20)) + \u0026#39;) for @Quarter = \u0026#39; + CAST(@Quarter AS VARCHAR(20)) + \u0026#39; has been cleaned.\u0026#39; GO In that case, I have got one stored procedure per partitioned table.\nBut there is no reason using dynamic SQL and create a generic procedure which would create cloned table automatically when needed.\nLet\u0026rsquo;s test the procedure.\n1 2 3 4 5 6 7 8 9 10 --Truncate selected partition: EXEC [dbo].[TruncatePart_BigOrderTable] @Quarter = 201702 --Check tables SELECT * FROM [dbo].[BigOrderTable] WHERE [QuarterNum] = 201702; SELECT partition_number, rows FROM sys.partitions WHERE OBJECT_NAME(OBJECT_ID) = \u0026#39;BigOrderTable\u0026#39;; That does work!\nVery efficiently and works for the older version of SQL Server as well. Great.\nLet me know if that helps you in any way or tell me about your experiences in similar circumstances.\n","date":"2017-08-20T22:00:15Z","image":"/uploads/2017/08/peach-2573836_1280.jpg","permalink":"/2017/08/truncate-partition-in-older-version/","title":"TRUNCATE PARTITION in older version"},{"content":" Introduction Dr Greg Low is well known in the SQL Server community, an internationally recognized consultant, trainer and author of a number of Microsoft white papers, blog posts. Greg holds a PhD in computer science, Microsoft Regional Director for Australia, a Data Platform MVP and first two MCM (Microsoft Certified Master) for SQL Server in the world.\nDr Greg Low is the CEO of SQL Down Under and hosts the podcast under the same title.\nOutside his work, Greg’s current passion is learning to read, write, and speak Mandarin Chinese.\nThis talk was recorded during the SQLDay 2017 in Wrocław, on Wednesday 17th May.\nWhat was the first version of SQL Server which Greg has started working with? What is the counterpart for Uber in China? And what does do one of his favourite companies in the world?\n\u0026ldquo;Ask SQL Family\u0026rdquo; podcast discloses all answers for these non-technical questions.\nTranscript Damian Widera: We’ve been talking to Brent already about the MCM and he told us that you’ve been the instructor. And he said that those three weeks was really great time for him to spend with the entire group and the instructor. It was the nicest time for him to work with SQL server and amazing people.\nGreg Low: Yeah, completely incredible group of people. And the thing that I liked is that I ended up probably teaching it about maybe eight times so it was about four times a year over a couple of years and the people who were in the early sessions they were mostly Microsoft people. There might be eighteen people or three people each time who weren’t working in Microsoft. Most of them were mostly premier field engineers from different parts of the world. And so each of the regions around the world was able to send somebody along and I had that person come. And then the rest was product support people. That were the main ones that were there. But it’s really interesting now because now I know so many premier field engineers and product support people and it’s pretty amazing where they all end up over the years. It’s great.\nKamil Nowinski: I just want to go back to the beginning and explain why we are doing this interview and for who it is. The idea is that we would like to show you not only as a professional but also as a normal, common person, to show your habits etc. Just to give people a different look.\nGL: I totally agree. When I do a podcast myself, I always get the people to say personal things.\nKN: What is your name and where do you live?\nGL: My name’s Greg Low and I’m living in Melbourne in Australia. I grew up in Brisbane in Queensland, and nearly everybody moves from Melbourne to Brisbane not the other way round so I’ve gone in the opposite direction.\nKN: Why like that?\nGL: I just end up getting remarried actually to someone from Melbourne.\nKN: So that’s the love reason, very common. And what are you doing for life?\nGL: What am I doing for life? Workwise I head up a company called SQL Down Under. Number of things we are doing now: the podcast and things like that and some of the tooling, we have – these SDU tools – a lot of people use those now. But the main things we do are mostly consulting work and mentoring. We do some trainings. I used to do more trainings but I try to keep it maybe to about one week a month maximum, because I like to be doing sort of real things.\nKN: Real projects?\nGL: Yeah. Because otherwise one of things I used to find is seeing a lot of people doing trainings all the time and the problem is that they end up in the situation when the only thing they know is what the Microsoft or the people making the product tell them. And the problem is that they often are wrong, but they don’t even know that they’re wrong.\nI also think that a lot of people have ideas on how something may be done but unless they’re involved in detailed projects that run over a long period of time, they don’t get to see the outcomes of their decisions. And I think it’s really important that when you are making decisions, you see the pain or the glory of whatever comes from those decisions. What’s very common in consulting companies is that people that come in make sweeping changes and make a mess and they’ll be gone before anybody realizes they make a mess. I don’t want to do that. So I think it’s really important.\nBut we tend not to do a lot of consulting. What I call consulting? I see consulting as going in and doing the job but not sharing a lot of knowledge with the people that are there. We tend to talk about mentoring instead. I would rather go and work with the people who are there, filling things where there can’t do things but get them to the point when they can do most of the things. Because in the end, when you leave, they are going to be the ones that are going to be there. So I think it’s really important to do a high degree of knowledge transfer while you are doing this sort of job so that the people who are there in the end can look after them themselves. That will sometimes involve training. That will often involve building things and showing them things and so on, or just architectural work. I do wish we did more architectural work because we end up doing a lot of remedial or fixing problems in the systems. The problem is that so many of these things could be avoided. But the type of problems I end up seeing are lots of performance problems and in many cases, it’s just poor design decisions that were made somewhere in the project and they could have been avoided.\nKN: Yeah, most of the issues come from the design problems.\nGL: It’s funny if you look at performance problems, I think about 70% of SQL Server design problems are in the application rather than they have anything to do with SQL Server.\nKN: 70%?\nGL: I would say 70%. I think that maybe there are another 20% which are indexing and those sort of various things. And then you start to get into the fairly odd things – blocking and locking issues. The thing is that a lot of places we go into, they will say: “We have this blocking problem” but what they really have is that they have an application problem with running way too many queries in the first place. Then they have something like the querying indexing problem when the queries are running for far too long and that’s why they have this blocking problem in the first place. So it’s not usually the actual blocking problem. And until I fix most of the application issues where we can, and then we fix most of the indexing and basics sort of issues, then it’s pointless worrying about blocking and locking issues because it’s just not the issue. If I’m talking to someone, look, this is where a lot of DBA’s end up being problem is that they’re often being able to change the application or look at it too much. And they’re trying to make a difference of something they have no control over it. That’s very hard, right? Because you might have a query and might say: How can I make it run faster? That’s important but far more important is to ask why are you running that query ten thousand times a minute? It’s a much more important question than how to make that query goes 5% or 10% faster.\nKN: And also there is a problem of the cooperation between DBA and DEV. So DBA’s monitoring the system and DEVs are fixing the queries and they are responsible for that kind of area.\nGL: I don’t think that you can fix a lot of performance problems unless you are capable of looking both into the development and the data side. It’s simply not possible.\nKN: How long are you working with SQL Server?\nGL: Since 1992.\nKN: It was 6.0?\nGL: 4.21 actually. On my laptop until recently I had a copy of version 1.1 running but the problem is it was OS 2 based version and I can’t make it run on the current virtual machines. Just a couple years ago I used to show people how 1.1 looked like – it was a really interesting look on their faces when they see it.\nKN: Yeah, it’s basically walking history book of SQL Server.\nGL: So it seems like a very, very long time. When I first worked with it, I didn’t like it and then I end up working with it ever since. But the thing is that it needed too much administration. Particularly even later in the 6.5, I was working in the software house and I wanted to use it as a database for our application. But you couldn’t just install it somewhere and leave it alone. It just required things to be done along on too much of a regular basis. SQL Server 7 was the first version when I just felt completely comfortable doing that because it would automatically expand files. The files in the operating system level where they used to have these devices. It was just horrible. It was a sort of a throwback to earlier ways of doing things. And 7 was the first version I used where we can put it out in an application and everything would just go quiet. And quiet is good if you work software.\nDW: As an example, you could have done online backups like you did to database disk and the database still remains online so that was the big difference.\nGL: Yeah, it was the first version – that 2000 version – end up being the main competitor to later versions of itself for a longer time. They would try people to upgrade people to 2005 and the main competitor was 2000 not Oracle because it was just very stable and ran well. The only reason I brought 2000 out was really to fix the collation issues that they had in SQL Server 7. In 7 you can only have a single collation for the database and it had to be the same one. The server had one collation right through. So the thing is that when you had a second application, if it needed a different collation, you actually needed another whole server because you could also stole one copy of SQL server on the machine as well. It was just a dreadful situation. And the thing that made it worse was that Microsoft had started selling the Great Plains software and it required the boundary collation and that was different to all the standard defaults and everything. So someone who was selling Great Plains would have to say not only “take your application”, even if you have a SQL server you needed to buy another SQL server because it couldn’t co-exist with something else. I probably think that internal pressure was the thing and then they introduced. I think what people see is the collation mess actually. A lot of the things ended up changing when collation start to become a problem actually. But it’s a flexible scenario what they built. The biggest hassle is the fact that the system has one collation system for TempDB and all the database is shared TempDB. When the database has all potentially different collations, they really should have a TempDB per database equivalent. And when you do contained databases it does that a bit. But again, they haven’t continued on with that. One of the things that I’m not keen on with SQL Server is that they often introduce features and if there isn\u0026rsquo;t massive amounts of interest immediately, they seem to lose interest in them and then the feature doesn’t get removed or fixed, it doesn’t get added to it, it just sort of drifts into nothingness. And the problem is that you have people come into product who don’t know the history of it. And then they’ll see something and they wouldn’t know the limitations and then they start investing in that architecturally like: “I’m going to use this feature” and yet everybody who works for the product knows that they shouldn’t be using this one. It’s not obvious for someone who comes to the product. They are going to continue with things until they work. Or they need to be get rid of them, or at least they publish and wrote that: “We’re not investing in this thing anymore” rather than not investing and not making it obvious. Because if I’m looking at contained databases, I remember when I was first introduced to the team that was showing it to us, they spent the entire session basically apologizing for what they didn’t get done. It was like they got some stuff done but that was nearly not enough. They were like: “Hey, look at this road map” and they had that long road map of what they were planning to do. But next year none of these people were there, the team wasn’t there, nobody discussed the road map anymore. There was just nothing. And I can’t see anything has changed since then.\nDW: How did you start with your conferences? When was the first conference you attended and then when you appeared as a presenter?\nGL: I have done technical presentations for a long time since about early 1980s. I was working with the HP and I was doing things with that. I was doing things with HP around products in HP. I ended up as a technical trainer within sort of an Australian-Asian Tech Support group. So I used to do a lot of technical training when I was there. Then I was running a software house so we have things that we built and then we had to train customers and so on. It has sort of grown from there. I think the first user group that I got involved with was a .NET user group.\nDW: You started at .NET user groups? [laughter]\nGL: I really enjoyed it there. The thing I used to like there is that I would go all the time, no matter what the topic was, because I would always learn something and it wouldn’t matter what someone was discussing. I remember a guy turned up and he was building some audio application or whatever, but then I sat there and like the user interface appealed to me. It was actually nothing to do with what he was building, because that part didn’t interest me. It was really interesting to look at the other things associated with it. And I always think that if you go to any of these groups and you don’t pick up something, you are not trying. And the other thing that I used to like to do when I started to run the groups I find it very important to try to grow new speakers all the time. Because I’d see groups all the time endlessly saying: “We can’t find someone to speak” and they have a room full of people and all of them are doing interesting things. Then they would be thinking that they have to find somebody to speak and this is not true. You have to have a process for getting people to start to speak at the groups. But the thing is for most people, if you go to them as say: “Look, I want you to speak on a fairly technical topic and I want you to talk for like an hour or something.” And they haven’t done it before in front of the whole bunch of people, they just go: “No.” They’re not comfortable doing that. So, I quickly moved to a model where we have two topics on every group meeting. And we would have introductory level topic for 15-20 minutes or something like that, then we would take a short break and then we would have the main topic. The thing that was really good with that was if you go to some experienced person and you say: “Look, are you prepared to talk about this simple topic for 15-20 minutes?”. It’s a completely different reaction that comes back. And what I used to find is that even then all of the experienced people in the room, you think they would be bored and senseless with the introductory topic, it’s not the case Because if you get someone who is fairly good and they research the introductory topic really well, like people actually do to do that 15-20 minutes, there always be something in there that even the experienced people would go like: “I didn’t know that!” But then people would get a little bit taste of doing that. Some of them would never do it again, many would go and do better things. I was really proud that in the group we were running, we were ending up sending out speakers to all over the places rather than having a problem with finding them. Because it’s just the case of trying to grow new speakers. I think this is one of the fundamental things that people running a group need to do. And the SQL server was an area of passion so we started a SQL server group as well.\nDW: And what about the podcast because you run them for a long time. How did you find that it could be interesting?\nGL: I just thought it would be interesting to have something where people could get depth on particular topic but also get to know the people who wrote the books, or in the product groups or things like that. So I tend to do normally three to five a year. I actually haven’t even done one this first half a year. We have a couple lined up in this moment. I just wanted to make it more informal but detailed thing where you can get to know the person and you can also get to know the topic. But I wanted people to have some background so that they not only understand how things works, but they know also more why it’s done. You need the people with the right knowledge and background to be able to fill that in. I remember having someone like Roger Wolter taking about Service Broker. There were just so many insights that someone who’s the product manager has that just nobody else has. They know why certain things are done. That’s a really good thing.\nKN: From the personal point of view what is your hobby?\nGL: Curious one – it’s learning to speak Chinese. Look, when I was at school I did Japanese for five years and I have worked in Japan for short periods. And I still never got that comfortable with Japanese. I could read hiragana and katakana but I had limited knowledge of kanji characters, these sort of Chinese characters that come from there. And that’s really restricts what you can do, because there are just so many characters to learn. When I was in Japan, I was ok with basic greetings and I could get by. I could read the names on the stations and things like that. I loved working in Japan whenever I was there. It was really good but I always wanted to learn Mandarin. If I look at language trends, I was looking at trend even just a year or so ago, they were saying that if you look forward by 2023, if you look at the native of people of any language, Mandarin is on top by so far, it’s not even funny. There is a massive gap between it and the second one. And the second will be Spanish which is surprising because people would think it would be English but it’s not. And English is just the third. I think that the problem is that the English-speaking world doesn’t understand that’s how things are. But what they are saying is that a big part of this is based around birth rate and right now one in four children born today will be native Mandarin. So get used to the idea. And one in five children born today will be native Spanish-speaking. If you look around the world – like South America and areas – these are the people with large Catholic families – eight to ten children. This doesn’t happen elsewhere in the world. They are just producing enormous numbers of people. In the case of China, it’s interesting because they have about nine hundred different dialects of Chinese and so Chinese it’s already way on top of anything else. But particular dialects – more or less Cantonese in Australia – was very common for people who came because originally they all came from Hong Kong. So, if I was sitting with them on train, I would here Cantonese all the time before. Now I hear Mandarin. And it’s simply because of Chinese government. If you are managing billions of people you just have to be consistent and they are getting to the point. They are just saying: “Look, that’s the language. Get over it!” I feel sorry for the Cantonese people but the way I look at that already the kids in the areas like Hong Kong are being taught Mandarin from kindergarten. And by the time they get a little bit older, every business thing they’re going to do or every government thing, it’s going to be more and more Mandarin. It may take a generation or two, maybe three, but I haven’t had the slightest doubt that the long-term plan for China is that the whole country would speak the same language. And already countries like Singapore, they already big languages too. Singapore is an interesting one because they have English as the primary language yet two thirds of the people are native Chinese so you just start to wonder if that’s a long-term plan.\nKN: Yeah, it looks like that. [laughter]\nGL: For now when I go to Singapore it’s interesting because suddenly it’s like there’s another whole part of the world opened up to me, that I didn’t used to see. I spent a lot of time just reading sort of Chinese technology things. I’ve tried to spend some time there in SQL Server forums and trying to answer some questions. That’s so hard. I look at their technology. In the western countries, we just never come across it. And they are making such advances so fast and it’s so not visible to us. You need to start watching it because it is really coming. Like one of my favorite companies in the world is the company in Shanghai (they are not from there but they are also in Shanghai). They are building the skyscrapers. And they are doing three storeys per day. And they recently build a fifty-seven storey skyscraper in nineteen days. That is so far removed from what anybody else does. And their thinking is that you should build these things in the factory and then bring it in like a big mechano set. This is what they are doing. What interesting is that even part of the building like the power cables are already in there. And as they are assembling it, all the bottom floors are already lit up.\nKN: And they are building the buildings like Lego blocks.\nGL: And there is as well a company which was currently printing 3D office blocks. Now they are printing 3D French mansions. Who does that? They are for about one hundred and sixty thousand US dollars! This is very peculiar. Everywhere I look it’s like all those companies, we still look at payments and PayPal is pretty big and global. But Wechat for example and the various sort of messaging applications in China – like one of those did more transactions in one weekend last year than PayPal did for the whole year globally. Right?\nKN: Wow! Another great number!\nGL: So, you just go: “That’s extraordinary!” Yet it’s like we are not seeing these things coming. Uber is considered to be disruptive around the world but Didi Chuxing in China was already seven times the size of Uber there.\nKN: Yeah, the economy in China is huge.\nGL: Do I think that we will see Didi Chuxing coming globally? Yes! Yes, I do. They’re been buying Lyft in India. But if you look at the parents of these companies, they are companies like Alibaba who had the largest IPO (initial public offering) of all the time. These are massive organizations. You cannot ignore them. These things are coming and people aren’t watching what’s going on in China. We were very divided thing years ago and we didn’t sort of interact that much. It’s interesting like years and years ago the USA used to worry about Chinese communism but now they are worried about Chinese capitalism.\nDW: It looks like a different story.\nKN: Let’s talk now about your work life balance and what daily habits you have.\nGL: At the moment, I had decided I never wanted to be going to like a cubical in the city and doing the same thing every day. Yet lately I’m going to a cubical and it’s just because I’m doing endless mentoring work at the moment on some longer projects. That is actually taking enormous amount of time. We still have these podcasts that I do occasionally. We are doing a lot of work on the tools. We’ve been building these SDU tools, and we’re just getting more and more tools and putting that in place. We have a whole online training – we are building that in the moment and this hasn’t been released yet but that’s sort of upcoming. And I’m trying to make sure that it’s a kind of slightly different experience. Most of the training that you have down there at the moment is sit-and-watch- a-video type but I’m trying to have something much more immersive because we still do quite a bit of in person training and that goes great. But I would love to get closer to that experience when somebody is doing that online from somewhere else. We’re working very hard on the things that we have to do to make that a reality. Because it’s one thing to sit and watch something but doing it is much more different story. So we’re trying how to embed that really nicely. We’ve also spent an enormous amount of time on the quality of the lab, content and things that we have. Again, we need just to repurpose that as a way of being able to do the same things online.\nKN: How many hours are you working per day?\nGL: A lot.\nKN: But the definition of “a lot” is different for people.\nGL: I do try to have eight hours sleep in a day. I didn’t have it last night but I try to do it. Because otherwise I get sleep deficit thing that make me hopeless by the end of the week. But it’s hard to tell who much time do I work per day because a lot of the IT things are my hobby as well as my work. And I think that’s one of the challenges that if you actually love the technology, it’s very easy to sort of mistake of what is work and what is not.\nKN: Then you spend too much time on your hobby or on your work.\nGL: And you can try and convince yourself it’s work but it might be just something that isn’t. I did a lot of work on our websites and built things and things like that. And so to do that I had to make sure that I learned HTML5 and CSS3. Do I really need to know that? No, not really. But now I have an interesting appreciation of CSS for example. I can’t say I like it. Because you can get amazing things done in a hurry, and then you can waste enormous amount of time trying to do something look the same on different browsers. And I think: “Yes, I can just hire someone to do that.” But the other thing is that I like being across that or maybe angular and love these things simply because in the work I do, I end up talking with the developers all the time and I don’t want be the best CSS developer but I want to know what they are talking about when they’re discussing it. Because then it puts you on a different position when you are talking to them about the code. And, also some of the sites that I’m in just have solid enough background on either VB or C#. Just to be able to dive into their codes and find things to be able to do a little proof of concept or basic changes or just to demonstrate what you are talking about, I find that’s really important. Whereas if I was just limited to doing the data, I would not get the same outcome. Giving you the example, a company is doing work for Melbourne who had the Microsoft software and Development Center in Sydney had built an application for them and it was so slow. But I would say it took us probably three or four months, we had it down to maybe one fortieth of the requirements of what it was when they had it delivered. They could buy forty times as much hardware. We can spend some effort and try to make things better. But the thing is you couldn’t do that just by tuning the database. We had to go back in through their problem to the bits of code. But the problem is that I find there are things you can do change the application, no question. But you can\u0026rsquo;t do that, you\u0026rsquo;re always going to be limited to marginal improvements.\nKN: Is that your first visit in Poland? Do you like it?\nGL: Yes, it’s great. I have spent time in Germany before so I’ve been nearby. I was looking forward to it.\nKN: What do you like the most?\nGL: I’m not sure. Just the people actually. The weather has been great too at the moment.\nDW: Yes, we are very lucky.\nGL: When you said that it was in May, I thought this is good because I’ve spent time in the UK in May and it was great. Because being in places where is cold and snowy, it’s not my thing. It can be nice. I remember that Miracle used to run conferences in Denmark and they had it in the middle of winter and it was down at La Landia, so it was a water park. So, a water park in Denmark in the middle of winter. Anyway, it was kind of interesting.\nKN: Greg, thank you very much indeed. It was big pleasure for us.\nGL: Thank you, guys. It’s been great to come.\nUseful links: Greg Low\u0026rsquo;s company: SQL Down Under\nGreg Low\u0026rsquo;s Twitter: @GregLow\nGreg Low\u0026rsquo;s Blog\nGreg Low\u0026rsquo;s podcastSDU Tools\n","date":"2017-08-12T13:13:04Z","image":"/uploads/2017/08/KNDW_Interview_with_GregLow_1600.jpg","permalink":"/2017/08/asf-002-greg-low-interview/","title":"ASF 002: Greg Low interview"},{"content":"Hello SQL Folks!\nI’m happy to announce we Microsoft released RC2 of SQL Server 2017. This release contains bug fixes and performance improvements\nI bet you know that SQL Server 2017 = SQL Server on Linux but there are more than only this. Of course, the ability to run it on Linux is the most welcome feature.\nLet me name the top 3 feature I really like the most:\nresumable index rebuild automatic database tuning new graph database capabilities Take a look at other features. The complete list is here.\nYou can try the version now.\nBelow you can find some useful links:\nHow to start with SQL Server on Linux Build an app using SQL Server SQL Server on Linux Introduction to SQL Server on Linux for SQL Server Administrators - Red Hat Introduction to SQL Server for Linux Administrators - Red Hat You can find great summary of what has been improved in this release on the official SQL Server Blog page.\nCheers, happy playing :)\nDamian\n","date":"2017-08-02T21:25:04Z","image":"/uploads/2017/08/P1044408.jpg","permalink":"/2017/08/sql-server-2017-release-candidate-2-rc2-is-available/","title":"SQL Server 2017 Release Candidate 2 (RC2) is available!"},{"content":"Hello SQL Folks!\nI’m happy to announce we Microsoft released RC1 of SQL Server 2017. There are lot of cool features in this release. I bet you know that SQL Server 2017 = SQL Server on Linux but there are more than only this. Of course, the ability to run it on Linux is the most welcome feature.\nLet me name the top 3 feature I really like the most:\nresumable index rebuild automatic database tuning new graph database capabilities Take a look at other features. The complete list is here.\nYou can try the version now but I can promise that it is not everything :) and some other things will come into the light soon.\nTwo very useful links:\nHow to start with SQL Server on Linux Build an app using SQL Server You can find great summary of what has been improved in this release on the official SQL Server Blog page.\nCheers, happy playing :)\nDamian\n","date":"2017-07-17T23:58:08Z","image":"/uploads/2017/07/P1004325.jpg","permalink":"/2017/07/the-rc1-of-sql-server-2017-has-been-released/","title":"The RC1 of SQL Server 2017 has been released!"},{"content":"Dear SQL Server fans :)\nYou can still submit a session for the upcomming SQL Saturday in Katowice that will take place on 16th Septmeber\nVisit the page and send the submission now. Just to remind you - the deadline is on 18th July!\nBy the way there will be four workshops and two of them will be in English:\nMikael Wedham - A master class on workload tuning (ENG) Stephanie Locke - Introduction to R (ENG) As usual the workshops start on Friday at 9 a.m.\nIf you speak in Polish than you should definitely take part in one of the two workshops:\nTomasz Libera - SQL Server Integration Services w praktyce Hubert Kobierzewski - Kompletna hurtownia danych w chmurze – krok po kroku Don\u0026rsquo;t waste time and register to the event now here.\nWe have already prepared and prepared the event schedule. There are some empty slots but even now you can see how great the event is going to be.\nThe SQL Saturday Katowice is organized as usual by the passionates from DataCommunity.\nI am proud to be one of the members!\nCheers\nDamian\n","date":"2017-07-13T21:09:21Z","image":"/uploads/2017/07/room-2192484_1920.jpg","permalink":"/2017/07/sql-saturday-664-in-katowice/","title":"SQL Saturday #664 in Katowice"},{"content":"Yeeaaahh! First post. First blog. I was waiting for it for at least 2 years. Yup, undoubtedly a \u0026ldquo;waiting\u0026rdquo; is the best definition for that. Waiting, because I was doing nothing towards this\u0026hellip; apart from a think about it.\nAlways was something else more important. But now - it appeared. Eventually. After many attempts, gigs of ideas, lots of thoughts and unrealized plans. Until today.\nAlso, the date of publication is not an accidental. It\u0026rsquo;s an appealing number: 07.07.2017, isn\u0026rsquo;t it?\nYes, I know. It\u0026rsquo;s only my exciting. For now. I hope it will influence to you soon as well.\nFine. Calm down Nowinski, calm down. What right now? How will the next steps be?\nI realise that that was only one part of work. The easier one, I guess.\nWhat will this blog be about?\nSQL Server and all related stuff! About everything I\u0026rsquo;ve been learnt so far, I\u0026rsquo;m experiencing every day, I\u0026rsquo;ve been read, seen and felt during many events I have attended.\nI hope that very soon I will also write about new features in terms of data (platform, analytics, processing) itself: U-SQL, Data Lake, R and Python, hence not only on-premise edition will be on top.\nWhat will the blog be for me?\nOhhhh\u0026hellip; Many things. Set of materials, archive, relations from events, links to other blogs, a list of interesting books and so on.\nBasically, this blog will be a record of my professional life. My passion.\nWhat will the blog be for YOU?\nYou tell me. Depending on your stage in career, experiences and courage - this blog might be a source of valuable information, can inspire you (hopefully), can be a field of comments and discussions or \u0026hellip; simply, another useless website which you don\u0026rsquo;t have time to read.\nHow can you help?\nYou did it already. If you\u0026rsquo;re reading this post - that\u0026rsquo;s great. Do read next.\nDo want to help more? Put your comment underneath. Tell me what you like, what don\u0026rsquo;t, whether do you agree with a post, discuss if don\u0026rsquo;t, share your experiences. Furthermore, do not hesitate to share these posts via Twitter, Facebook, LinkedIn\u0026hellip; wherever you want. That would be a prize for me and a motivation for further to work.\nHave a good read. And journey! Like mine.\nKamil Nowinski\n","date":"2017-07-04T01:00:01Z","image":"/uploads/2017/07/railway-2439189_1280.jpg","permalink":"/2017/07/new-blog-first-post-big-challenge/","title":"New blog, first post, big challenge!"},{"content":" Introduction Brent Ozar talks about himself, his passion, hobbies and carreer.\nBrent Ozar is one of the most known person in the SQL Server world. He is a Microsoft Certified Master (MCM) and Data Platform MVP. His passion and energy is shared among all people he work with. He is known as a very energetic speaker. He loves sharing his knowledge. He loves cars. He loves dogs.\nThis talk was recorded during the SQLDay 2017 in Wrocław, on Wednesday 17th May, 8 am.\nTranscript Kamil Nowinski: Yeah, great. That’s my first interview.\nBrent Ozar: Ha, ha, awesome!\nKN: I’ve never been working as a journalist!\nBO: Oh, I think it’s so much easier when at least you’ve talked to the person before. You’ve spoken to them and at least you know you’re not nervous or worried because it’s\u0026hellip; If I had to go and speak to politicians or something like that, oh man!\nKN: Yeah!\nBO: Or celebrities\u0026hellip;\nKN: Yeah, exactly, that would be even worse, too stressful. But in that way, I’m not stressed because we are simply a family, so there’s a great approach, everyone has a great approach. Everyone is very friendly and that’s amazing.\nBO: As opposed to if you’re trying to interview a celebrity or a politician that always has something to hide and you’re trying to get it out of them, like trying to out-think: “alright, how am I gonna get this person to talk about their policies or whatever that’s gonna\u0026hellip;”, and they’re trying to battle it back. Oh, I could never do that! Very tricky.\nKN: Yeah, that would be difficult. Ok, how was your night?\nBO: Wonderful! Very good. It was neat to see so many people out there. I was like 60 seconds late to the group photo, the first one, the huge one.\nKN: Yeah, but I saw you on the photo so you were there.\nBO: You mean in the speaker when I was there, in the speaker when I was on time. But in the huge group with all the attendees, I was working my way up through the back, ‘cause I’m like, I thought I was looking at the time, I’m like: “oh, still got another five minutes”.\nKN: Yeah, I heard that.\nBO: Oh my gosh\u0026hellip;\nKN: I was a couple of minutes before it was seven, or something like that, 7 pm. But I saw you on the picture that Paweł Potasiński took.\nBO: Oh, yeah!\nKN: So he just took the picture or something like that\u0026hellip;\nBO: Oh, cool, yeah.\nKN: I will send this photo to you as well.\nBO: Haven’t seen any of them yet, so\u0026hellip; It’s been a zoo. Thought it was a nice party. Nice talking to people.\nKN: Yeah, exactly. I think it was a very good conjunction with the knowledge and fun, so every time it’s good to relax after a lot of knowledge about the whole day. A lot of sessions.\nBO: Yeah, you must be happy too, because everything looks like it’s going great!\nKN: Yeah, yeah! Everything looks good so far.\nBO: Touch wood, but yeah\u0026hellip;\nKN: Unfortunately, my laptop didn’t work yesterday. I tried to connect my laptop to the room, when I was showing my session today, and it doesn’t work. I mean, the laptop works, but the duplicate screen doesn’t work.\nBO: Noo!\nKN: So I just tried to fix that problem yesterday and I spent about 19 minutes and I didn’t fix it\u0026hellip;\nBO: Oh, exciting!\nKN: So today I will check two other tricks and if that doesn’t help, I will need a laptop from some guy.\nBO: I’m guessing you do demos? Or you’re doing demos with like SQL Server or is it only PowerPoint?\nKN: No, its PowerPoint, 50/50, so\u0026hellip;\nBO: If it helps, I have spare laptops. It’s SQL Server 2016, if that works.\nKN: Do you have Replication there?\nBO: The replication?\nKN: The old one, yeah. The old SQL Replication [laughs].\nBO: No! [laughs]\nDamian Widera: So you don’t have the component installed.\nBO: Wow, well, yeah, that’s a problem.\nDW: No, I think we can manage that because we have laptops.\nKN: Yeah, we will.\nDW: I already offered Kamil a laptop but I have to leave right after the lunch and Kamil has his session then.\nBO: Well, there will be enough presenters, so somebody will have\u0026hellip;\nDW: Yes, we can do this but it’s very interesting that the audio does not work. I mean the video does not work.\nKN: But I guess it’s the fault of my laptop because I had some similar issues at home, so yeah.\nBO: You should check with this Greg Low, too, cause I bet Greg has Replication on his. If he’s not in the same time slot, I bet he\u0026hellip;\nKN: No, no worries, we can manage that.\nDW: You should start early. [laughs]\nKN: At least one of them will give me the laptop.\nBO: Yeah, wow.\nKN: Ok, let’s start! I hope that we’ll have some questions from you as well! [laughs]\nDW: I have some in my head, so\u0026hellip;\nBO: You have to put a piece of paper in the middle, then he can steal your questions. [laughs]\nDW: But you can start.\nKN: Of course, yeah. So Brent, thank you for accepting my invitation.\nBO: My pleasure!\nKN: As I mentioned, this is my first interview, so\u0026hellip;\nBO: Ta-daa. I’ll go really easy on you. [laughs]\nDW: Sounds like we’re trying to hire you to\u0026hellip; [laughs]\nBO: Ha, ha! Oh, that’s not gonna go well. I’ve had so many bad interview stories!\nKN: I saw that kind of interview on your blog, so\u0026hellip; where you’re explaining people how to ask people if you want to hire them.\nBO: Yeah! Especially because I get to a point in my career where I don’t wanna do job interviews again. Like I feel like I never wanna go to work for anybody, I have my own company. But that’s when you do really well with interviewing, when you’re confident and relaxed. When you’re mellow. And so many people go out try the interview for a job, and they’re freaked out, they’re worried, and it comes through. If you can just be relaxed and be yourself, you’ll make world of difference.\nKN: Yeah, true. So at the beginning, I would like to say you shouldn’t worry if some questions are simple because I just want to prepare this article, this podcast for people who maybe don’t know you very well, or also, this interview\u0026hellip;\nDW: We have people who do not know Brent?\nKN: Probably not!\nBO: My parents\u0026hellip;\nKN: Sometimes there are new people on the market so it would be good to introduce Brent to them, and allow them to start working with a very good tool from Brent like Blitz, yes?\nBO: Oh, cool! Yeah, that’s true.\nKN: But also, I want this interview to show you as a person, not only as a professional. From another side. Like a usual person, your ideas for life and\u0026hellip; So the first question is: what is your name and where do you live?\nBO: My full name is Brent Gregory Ozar and growing up in high school, I always went by Greg, cause my Dad goes by Greg, so I wanted to be just like my dad, so it’s Greg, Greg, Greg. And in college, when you get to start over when you go away from home and I went: “Ok, I guess I need to be my own man now”. And so I started going by Brent again. But in America, I never use my initials, Brent Ozar, because BO stands for body odour in America, so no-one wants to use those. So I was going by BGO. Right now, I’m in down town Chicago, but we telecommute full-time, so I move\u0026hellip; It’s been every few years, so I’ve lived in, throughout my life, Kentucky, Indiana, Illinois, Texas, Florida, Mississippi, Missouri, all over the US. And I really love travel and seeing new places.\nKN: And currently, where are you living? I know that but you know, just wanted to\u0026hellip; [laughs]\nBO: Down town Chicago. My wife is from El Salvador, she grew up in El Salvador and had never seen snow before! So at one point I said: “let’s go up north so you can see what snow is like”, cause growing up in Michigan, I know what it’s like. I thought one winter would cure her and she would never want to see it again. We had 10 feet of snow in the first 40 days, and she loved it. She couldn’t get enough of it! But she missed the big city, so we moved to Chicago instead of Michigan and we totally love it. The winters suck but\u0026hellip;\nDW: For us, but maybe not for her.\nBO: Yeah!\nDW: I hate winter too because I have a lot of work with the snow, so\u0026hellip;\nBO: Yeah, yep. We have a little small dog, and I have to put boots on the dog, and all that, yeah.\nDW: So how is Ernie going?\nBO: Very well!\nKN: Ernie is your dog?\nBO: Yes!\nDW: I’ve seen a lot of pictures.\nBO: My hairy daughter.\nKN: On Instagram?\nDW: On Instagram.\nBO: She’s just the love of our live. My wife has never had an indoor dog before and I’ve always had dogs, so I got her finally to have a dog. But she has sworn that once Ernie passes away, that’s it. No more dogs, no responsibility, just goof off and travel all the time.\nKN: That was the trade-off, yeah?\nBO: Yeah. Cause when I leave, I come to Poland, I’m going to Florida next, she’s at home with the dog. And she’s like: “eh, I don’t wanna go outside five times a day”. I’m like: “OK, I can understand that”. I don’t like responsibility either.\nKN: Another simple question: what do you do for the living? In a few sentences.\nBO: I love making SQL Server faster and more reliable. When I was growing up as a\u0026hellip; I worked at hotel, I worked in restaurants, and that’s too hard of work. You’re working continuously, you work every holiday, you work every weekend, because that’s when people wanna go to hotels and restaurants, and I said: “I’m done with this, I gotta find something easier”. And I went into IT cause I’ve always loved computers. And I didn’t realise in IT, you’re on call, and you work every weekend, and things break every holiday. But I’ve always had the strategy of “I wanna stand near whatever’s most expensive and wait for it to break”, cause that’s the fun part of the business, you learn a lot.\nKN: So you like it. [laughs]\nBO: Yeah, SQL Servers, they’re expensive part of the business! And everybody in the business interacts with it, you get to work with sales people, developers, IT executives, so it’s just so much fun to be near SQL Servers that are breaking, and figure out how to fix them.\nDW: But how did you find that SQL Server is interesting? There are a lot of servers. You could’ve ended up with Exchange for example, yes?\nBO: Yeah. It’s not expensive enough. There was even a point where I almost went into Oracle. I had a friend of mine who worked in Oracle. But at that point, I had already learned so much about SQL Sever, and it looks like a black box, but the more you work with it, the more predictable it is, and you understand how it works. And the community is so good! The first time I found Pass, the whole SQL Server community, I went: “Wow, there’s actually like a family here of people who talk to each other, they share things”. And it’s not that way in all other servers. A lot of other servers, people hoard their knowledge, they don’t wanna share anything, they’re afraid. And it’s so different here. So it’s been like coming home.\nKN: So when was your first conference? So you have been as an attendee, and at some point in time, there was a switch and you were a presenter.\nBO: I wanna say in 2006, I went to my first user group meeting. It was either 2005 or 2006. I went to a Microsoft office in Florida, Fort Lauderdale, Florida, and I sat in a meeting, and I remember the presenter, and he wasn’t a great speaker, but he was sharing what he knew. And I went: “Oh, I’m not a great speaker, I can share what I know, let’s do this!” And so like from the first user group I went to, I went: “I wanna be up there on stage, cause I have that I can share, mistakes that I’ve made”. Within that year, I went to my first PASS summit, because my boss said: “We have a training budget, do you want it?” I’ve been working in IT for like 10 years, I didn’t know there was a training budget! You know, I just got into the right company where I can go to a conference? Are you kidding me? So I went to the summit, the first summit I went to, and I couldn’t find anyone to even have lunch with! I just didn’t know the right questions to ask, I didn’t know who to go up to, I didn’t know I could go talk to presenters. It was terrifying! And the second time, I got to know more and more people, I think the third time, I know I applied to speak and I got turned down. And I knew Kevin Kline at the time, and Kevin happened to email me and say: “Hey, we have a presenter drop out”, while I was there, he said: “Could you jump in and do a session?” And I went: “Sure! I’d love to!”\nKN: Wow, that enthusiastic?\nBO: Yeah! Why not! Because I had spoken at a few user groups, that’s the way you want your first PASS summit speaking session to go. There’s no pressure. I didn’t have to worry about making everyone perfectly happy cause it was better to have a session than none at all! I was just happy to be up there. So I got really lucky there, and I think I’ve been speaking every year since then. I think I missed one year switching schedules, but it’s just been addictive. Once you get up there and you see the light go on and people’s eyes, it’s so rewarding.\nKN: OK, so your first user group was in 2006, yeah?\nBO: Sounds about right. Either 2005 or 2006.\nKN: And what was your first contact with SQL Server?\nBO: 1998–99, I was a sysadmin at a hotel company, and the accounting software was in SQL Server. And my boss is like: “Hey, this thing breaks every now and then, can you go and fix it?”, “Sure, let me go figure it out, how hard can it be”. And then once I understood how a table worked, and how you could write SQL queries, I started playing around with Microsoft Access, but I wasn’t happy with that, I switched over back into the SQL Server side. And I did some web development with Macromedia Dreamweaver, it was this old Ultradev\u0026hellip;\nKN: Yeah, I remember that too!\nBO: We’re all old\u0026hellip; Dreamweaver Ultradev. And we’d make little webpages for hotels to enter in their sales numbers every day, and it would go into our accounting system. So I dabbled a little in 6.5 but got much more into it when 7 came out. Like I don’t remember anything about 6.5 but I liked 7 a lot.\nKN: What kind of specific tools are you using during your professional work?\nBO: I have the weird job ware. As a consultant, I usually only have to fix one server. So I don’t do the same thing a lot of DBAs do. A lot of DBAs would get into PowerShell, cause you need to manage fifty servers or a hundred servers. If I had a full-time DBA job now, I’d be using PowerShell. I’d be using Visual Studio code, coding my PowerShell inside there. But because I have the weird consulting job, where I only get one server at a time, it’s just Management Studio, PowerPoint, and Excel. And just back and forth between those things. I have a weird mental competition thing, where I tried to get better at stuff, so these days I’m trying to code my T-SQL in a text editor with no IntelliSense, to see if I can figure out if I know the joints and the DMVs and all that.\nKN: Wow, that’s the challenge.\nBO: Yeah, why not! Just to see. Intel script stuff as far as I possibly can dare, and then I go: “OK, now let’s go check and see what I’m doing”. And then I’ll switch in the Management Studio again.\nKN: Like in the old school way when we just wrote our code in C++, C+.\nDW: Or webpages in Notepad for example. Simple tools.\nBO: And I think the other thing that’s tricky with that is I have to write a lot of my own scripts, only because a lot of the time clients won’t let me install stuff. I love third-party tools! I mean, third-party tools are amazing. If I had a full-time job, I would buy all third-party tools immediately! But they usually sit me down in front of SSMS and say: “Go”. So I write a ton of scripts, that’s where the sp_Blitz, Blitzcache, Blitzindex comes in.\nKN: That would be my next question, how do you find that you should write your own procedure or diagnostic tool and how did you decide that you would like to share this tool with everybody.\nBO: On GitHub.\nDW: Because that’s the point, that you have your knowledge and you would like share the knowledge among all other people.\nBO: When I first got started with this, I had this script that I would go run whenever someone brought me a SQL Server when I was a DBA. Because people kept bringing me horrible SQL Servers. They would say: “here, we just acquired this company, they have these five SQL servers, they’re yours now”. So I would go look for triggers, is there database corruption, is mail set up, is alerting set up. I had this big, long script, and I had to run it section by section, and read the results. So I gave this out to the public, and I did a PASS summit session on it. This was the original Blitz, before it was sp_Blitz. And I told people: “I need you to run it section by section and read the results, and understand what it means”. One of the scripts in there tested database mail, and the email address was mine. And I said in the script: “Change this email address to your own”. What I learned very quickly was no-one was reading the script. [laughs]\nKN: They just ran the scripts and\u0026hellip; [laughs] waited for the results.\nBO: Unbelievable! They just copy things from the Internet, and they run them in production! So I got emails every day from SQL Servers all over the world. And you could tell which companies they were from\u0026hellip;\nDW: So you must’ve been very surprised.\nBO: I was shocked! So I thought: “If this many people need this this bad, and all they’re gonna do is hit Execute, why don’t I give them a stored procedure that just gives them what they need and say, here’s the things you should go do on your server”. So that’s where sp_Blitz came out where I\u0026hellip; let me just give you a stored procedure and it will tell you what to do. And for the longest time, we kept it as a free download on the site, but it wasn’t open-source. It was copyrighted, because we would find that sometimes people would take the script and they would change the contents to make it look like it was from them. I was like: “That’s kind of crappy, I wanna be able to stop that in case it happens”. Over several years, I went: “I don’t really care if it does happen. If it does happen, that’s kind of OK, because if people can get their problems fixed, I can’t fix every problem, I can’t be everywhere”. And sometimes people need this for different purposes. They need to take out some of the checks, they wanna add their own checks, whatever. So we just talked as a company and went: “What if we just open-sourced the whole thing and call it a day? And then, if people wanna check in their own code, they can”. Plus, Microsoft has been slowly moving towards the dance floor with open source. And I went: “Why don’t I get experience with that and give it a shot?” And this community is so friendly and open-source welcome.\nKN: Yeah, many people want to be involved in that.\nBO: Contribute, yeah. So I just went: “Let’s see what happens”. And it’s been great! People checking contributions, they file their issues on GitHub. GitHub is not easy to use, it’s not friendly, it’s a pain in the rear. But it’s what we have for an open-source community. So I’m like: “OK”.\nDW: Open-source community loves GitHub and they use it all the time, so.\nBO: It’s the best thing we have, so.\nKN: But still, I see a lot of comments from you and from Eric, I can see a lot of his involvement in that project. And I was wondering when you have time for that, for everything?\nDW: Isn’t that that you are working at, for example you’re going to fix a server, and you think: “OK, I would need that script because maybe I can use it also in some other place”.\nBO: Oh, that’s exactly\u0026hellip; I’m on a project right now where the client wants to know if the server goes down, how much data are they gonna lose, and how long will they be down for. So I talked to him and said: “Can I write it as an open-source script? Can we just build it and\u0026hellip; you’re still gonna pay for it either way, whether we build it just for you, or whether we build it open-source. But the nice thing, if we build it for open-source, people will find our bugs, and help us fix them, and they’ll add more features for stuff that they want, so you’ll get a better product. And it’s not your business”. That’s not a software company that’s selling monitoring tools for SQL Server, it’s not like this is gonna make you more money, so they went: “Sure, yeah, why not”. So a lot of the time when we’re working with clients and we find something weird on the server, we try to build that back into these stored procs, so we can catch it next time, too. Another thing that’s weird with this, you might thing that by giving the scripts away, you’re kind of taking away business. What we find is that most companies, when they call us, they haven’t even run those stored procedures. Or if they did, they’re like: “I understand what this means”. This is too much work. Help me figure out the plan to make this happen.\nKN: It’s not about having the tool, you need to know how to use the tools. And that’s the knowledge. You are the knowledge. OK, let’s talk about your hobby. What is your hobby?\nBO: Oh, that’s a tough one. I have a few things that I love to do but the most devilish is watching TV with my wife. At 5 o’clock, the instant the day is done, I check out, I’m done, and I go watch some of the dumbest TV you’ve ever seen. I watch Survivor, there’s a lot of shows on Alaska in the US that I love, Project Runway, just all kinds of goofy things. But beyond that, I love travel, and I really love cars. I adore cars.\nDW: Yes, we’ve seen some pictures. [laughs]\nBO: Man, I just love cars.\nDW: I think the car must be really fast, you like speed too.\nBO: Yes, and even old cars, as long as they’re beautiful. And one of the reasons I love coming to Europe is seeing all the cars that we don’t get over here in the States. Even small hot hatchbacks, just things we don’t get the chance to get is amazing to me. Building a car is so much work, there’s so much engineering that goes into it, all this precision that people don’t realise. Especially with all the features everyone wants. Airbags, anti-lock brakes, it’s incredible technology to see how these things work. I love them.\nKN: So which car is your most favourite?\nBO: My most favourite is probably the one I just sold. I had an Audi RS6 and I loved that dearly. But there’s no reason to own that in Chicago. It’s all potholes, 30 miles an hour, just drives me crazy. So I made the decision that I’m not buying another car again until I get out in the country somewhere where I can just go hammer it, and then it’s a Porsche 911 Targa. I want a Porsche 911 Targa, I’ve always wanted one since I was a kid, it’s just waiting until I get the nice open space. Right now I have a Jeep Wrangler and an Infinity G35, which is like just a regular 4-door family car type thing.\nKN: Great. What do you think about work-life balance, what is your approach to that?\nBO: Oh, I have strict time limits on when I work. I’m a really early-morning person, so I’m usually up at 5, I walk the dog, have breakfast, and I’m working by 7 at the latest. But I’m done at 5. I’m sometimes done before that, just cause I’m kind of stupid in the afternoons. Afternoons are not my best time. And then at the end of that, at 5 o’clock, I’m not walking back into the office if I can help it. I’m just not interested. I’d rather spend time with my wife. But on the weekends, she and I kind of have an agreement that she’ll sleep in until 9 or 10, so on the weekends from like 6 till 10 is another little nice work, just some jump that I get into. Now where it gets awkward is my wife works for us now. She’s like our office manager, admin assistant, so when she has stuff that she needs to get done, she will work on the weekend. And then if she’s working, I am not working.\nDW: Do you remember the MCM programme, when it originally started?\nBO: Yeah, I do, I loved it dearly.\nDW: How do you find it? Was it a great programme? Because it was initially like the first round when you had to go to Seattle or Redmond\u0026hellip; Three weeks\u0026hellip;\nBO: There were two parts of the programme. One was: you got to be in Microsoft, at their offices, for 3 weeks, listening to people who were brilliant, asking all kinds of questions. And it was very hard. They gave you written tests every week, and if you failed any of them, you were out. You could still stay there, but you weren’t gonna be an MCM until you could pass those. And those 3 weeks were amazing because it was like a boot camp with other people who were really good at SQL Server. And there was no work-life balance in those 3 weeks. You were in the hotel, you were all studying continuously, making flash cards, telling stories about SQL Server. We built some really close bonds amongst the 15 or so people who were in each class. And it wasn’t a class that was going to take you to master level if you didn’t already come in with a lot. It had like a 70–80% failure rate, cause they were really hard tests. That was the first part. And then the second part was: at the end of the 3 weeks, you had a 6-hour lab, and this lab test was the closest thing for me to a real consulting engagement that I had ever seen. You walked in and you had three sheets of paper, front and back, just single space, crammed with a brain dump from the client. They were just talking about things, here’s our servers, here’s problems that we’re having. This wasn’t a test where it was question 1, question 2, question 3. There wasn’t even a task list of “you must do this”. You had to figure our what you needed to do, and then there were your VMs: “Go!”. There was no-one left behind to help you. It was just “Go!”.\nKN: So it was just like a description of a case scenario, or something like that?\nBO: Well, it was like: “We’re having this problem, this query is slow, this server seems to throw an error, we don’t know what’s going on”\u0026hellip;\nKN: And you needed to just start investigating without a computer? [laughs]\nDW: There were 15 questions and 6 hours or so\u0026hellip;\nBO: And they never really told you how many questions there was. So the first thing I did in those 6 hours was I took a piece of paper and I wrote down what I thought were the questions, and I came up with like 20, and I think there were less, because I know from one of them, one of them I wrote down was: the client had said: “We’ve never been able to get a back-up of this server” and in my mind, that is “Brent, we need you to set up back-ups”.\nDW: That’s an amazing question, yes? Do you expect something like this on the MCM exam?\nBO: It was tricky because I didn’t know what\u0026hellip; Real clients run into that all the time. They’ll just kind of casually say: “We don’t have back-ups”. And I thought no MCM would leave a server without having back-ups. And you were allowed to write a text file for the client to leave behind, with instructions, so I said: “I’ve set up Ola Hallengren’s back-ups, here’s the instructions on how to use them”. Although, I think in mine, I actually used maintenance plans, because my logic was: if a client isn’t smart enough to do back-ups, they’re not smart enough to troubleshoot all the scripts either. So I think if I remember right I said: “Here, there’s maintenance plans set up and all this”. I burned time doing this, and it wasn’t even a question. It was just something they left in there as part of the case. So at the end of that 6 hours, I even stopped early because you had three VMs, but all of the questions involved these three. You could fix one thing and break something else that you’d done earlier. So I was maybe 5 or 5.5 hours in and I said: “Do you know what, I’m not gonna get any better, this is about as good as it’s gonna get, and anything that I touch, it’s gonna get worse”. So I closed everything up, when out, and I was just\u0026hellip; my hands were shaking, I was on the verge of tears, I cried, called my wife, and I couldn’t wait to do it again! I was like, this is awesome, I love it, it’s like a videogame for SQL Server! And as the other students came out, we were all talking about what we thought the questions were, what we thought our answers were.\nKN: A real community, yeah? At a high level.\nBO: Yeah, very, very sharp. And you listen to those people, cause you really don’t know where you rank, where you are relative to them. And as I’m listening to some of them, and we’re talking about what we did, I’m like: “I’m screwed, there’s no way I pass”. And all of us are thinking that same thing, because it’s got a 70-some percent failure rate, you kind of expect it. And it took them a couple of days to grade it because there was so many. And when I found out I passed, I was just on top of the world, I was like: “this is amazing!”\nKN: So back to the question on the MCM exam, you could have expected that kind of tricky question because if you passed some exams before, I mean the Microsoft exams, every exam has a lot of tricky questions.\nDW: But this I think is a different level.\nKN: But I mean the Microsoft approach to testing people. [laughs]\nBO: Oh, it was bad. There were\u0026hellip; and I’m sure I can talk about it now, cause it’s so dead and gone, but for example one of them was you needed to set up mirroring between two servers, and you think: “mirroring’s easy!”\u0026hellip;\nDW: “I can do it in three clicks”.\nBO: Yeah, “I’m done!” And you go to set it up and it fails. It fails immediately. And you start digging in deeper, and databases were set up to auto-close and auto-shrink. There were the wrong encryption protocols on one of the servers. There was all this tricky digging you had to do, so when you write out there was 20 questions, you think: “Oh, 5 minutes per questions and I’m done, you know, I’m outta here, maybe 10”. When you catch one of those “just set up mirroring”, and next thing you know, you’re an hour into it and it’s still broken, and you’re like: “I gotta keep going”, so it was so tricky. Every one of the questions had even looked easy, they had purposely set up so many roadblocks to make it hard for you. I loved it! It was so much fun, so much fun.\nKN: So I already know you’re planning to move to California, yeah? What is the reason?\nDW: Windy City to sunny California.\nKN: Completely different one.\nBO: I’ve always felt like a California guy, I feel like I either belong to Texas or California, those are the two. And in America, they’re radically different states. Texas is a bunch of guns, and it’s a red state, all-Republicans. California is vegetarians that eat granola and it’s a very Democratic state. But they both have more in common I think than people realise. They’re very laid back, they’re friendly states, they’re welcoming, great weather. But I’ve always loved California just for the wide-open roads, surf, I love being around the water and beaches. There are earthquakes, yeah, but nothing can be perfect.\nDW: But Texas has tornados.\nBO: Oh, it does. Huge thunderstorms and lots of guns, so yes. [laughs]\nKN: Do you plan to expand your company? And which direction if so?\nBO: Yeah, about 2–3 years ago, I guess about two years ago now, when I bought out Jeremiah and Kendra, my two business partners, we were all on exactly the same page that we wanted to grow the consulting side of it. We wanted to get to like 10 or 15 consultants. So I started trying to grow it, and it’s really hard, because there’s a huge gap between having, say, 5 consultants and having 10 or 15 consultants. You need a human resources person, you need a whole sales staff, you need offices, there’s all these things you need. And it’s financially risky. I got a little ways into it and went: “You know what, no, I’m not doing this, this isn’t for me, I’m happy where I’m at with the consulting”. But we do plan to grow on the software side. There’s a bunch of things that I think would make DBAs’ jobs a lot easier if they had them. So one of the first things we brought out was paste the plan, where you can paste in an execution plan, and you can share it with friends, post on Stack Overflow to get help, cause there just wasn’t a good place to go see execution plans. I think there’s a lot of room like that for online tools for DBAs. We had a lot of thick client tools but there’s not a lot of tools to make it easier for us to collaborate together online. But that’s definitely where I wanna grow. I’m a huge fan of online stuff.\nDW: And this stuff is easy because you just have to use it. You don’t have to install anything, you just have to use it and that’s all.\nBO: You don’t have to troubleshoot on some of these computers, yeah. I’ve been a developer of apps before and debugging is horrible! Debugging is awful. People do crazy stuff with their computers. People run on old versions of Windows, people are behind crazy firewalls, people run it old tiny monitors or huge monitors. Debugging that stuff just absolutely sucks, whereas if I can have it centrally, it’s like a Software as a Service, it’s way easier, much easier.\nKN: And this is the way the entire world is moving, we’re going to services. So I would ask another question, what would you say to people who wanted to start to be like you? Because you know, people, most of them, are here in Wrocław just to meet you\u0026hellip;\nBO: No! [laughs]\nDW: That’s right! To meet you, to attend your session, attend your workshop. And most of them would like to be like you. “I’d like to be Brent Ozar”.\nKN: Currently you have a lot of idols [fans].\nBO: Start drinking. Drink a lot. [laughs]\nDW: Apart from this. That’s not the message we’d like to share with them in the first place. [laughs]\nBO: The biggest thing is start a blog, just start writing down. And it’s so intimidating, because feel like you can’t share things, like it’s already been written somewhere else. Everything has already been written, that’s what books online is for. Everything is already in books online. But it’s how you bring your own personality to it. People don’t wanna read books online. Books online is deadly boring. How can you talk about what you do, because it took you months or years to figure out whatever it is you’re working on today. If you can share that, there’s hundreds of people or thousands of people behind you. I always think any IT industry is like a pyramid. There’s very few people at the top, with incredible skills, Paul White, Paul Randal, Kimberly Tripp, Kendra Little, there’s all these people who have unbelievable skills at the top, and we all wanna be like them. We all think: “Oh, I should write something that they would wanna read. They don’t wanna read my stuff! That’s not where they get what they are. Paul White is running a debugger to learn what he learns”. You wanna think about the masses of people who are under you in terms of levels. There’s always more, no matter how junior you think you are. Even if you’ve only been working with SQL Server for a year, there’s someone who has never worked with SQL Server and they just got handed their first set of queries to go build. So just blog about whatever it is you tackled today, start sharing that online, and it’ll seem like no-one’s reading in the beginning. I write blog posts that no-one leaves comments on today, that’s just the way it works. But over time, it snowballs together, more people find you via Google, and then when you’ve written enough stuff, then it helps you get up to start to present about that. When you start seeing comments from people saying “Thank you, that helped me”, then you realise you could stand up and present about it too. Every time you’re sitting in a conference, just like SQLDay here, you think: “I must be different than the people up on the stage, I’m not good enough to be up there” and all of us on the stage feel the same way! “Why am I here?” You go into a speaker room and you’re surrounded by these people that you’re like: “Oh, that person knows more than I do, that person knows more than I do”. At this conference I see Greg Low, Greg was one of the instructors in the MCM programme. He knows so much it’s unbelievable and he’s the most humble and down-to-earth guy you’ll ever meet. So it doesn’t matter how far you get in the community. There’s always gonna be people you look up to. Don’t look up in the pyramid, look back at all the people who need your help, and that’s what it’s about is how can you share that and put your personality in it, so the people start to know you and relate to you. Don’t write generic white papers, cause no-one bonds with white papers. They bond with you.\nDW: No-one wants to read it. So the idea is you should help others.\nKN: Share the knowledge.\nDW: And not want to impress the likes of Paul Randal, because it could be very hard.\nAnd you’ll be wrong. [laughs]\nDW: I expect it’s hard for everybody but it’s especially hard for people who’d like to start. So it’s not possible to impress a guru like Paul, like you or other people. And the other thing I think people are scared is that “what if I wrote a blog and somebody would hate it”.\nBO: Ha, ha, yes!\nKN: Hate across the whole world.\nBO: Delete bad comments. My recipe for that is, go to any YouTube video, go to a music video that you really love, or scenes from a movie that you really love, and start reading the comments. And people hate everything! You’re stupid, you’re ugly, you don’t know what you’re doing, you should’ve done that differently. You just can’t let that bother you. Everyone will always be a critic. And the other thing people are often scared of is “I’m gonna say something that’s wrong”. Well, of course you will! We all make mistakes! And then you just learn from that. You can edit your post! A lot of times in our posts you’ll see “Update” at the end, “here’s some things that I learned later that changed how this works”. Nobody’s perfect, you just start getting on that bicycle and moving forward, you’ll crash a few times, but then next thing you now, you’re having a good time.\nKN: Sometimes you think that for some people, hate is their hobby. [laughs]\nBO: Yes! They sit in their mother’s basement and downvote everything. Yeah, it’s amazing how there’s a lot of negativity out there, you just can’t let that bother you.\nKN: Let’s do another question at the end of our chat. How often do you fly to Europe?\nBO: I try to get at least once a year. I try to go to a different European country every year, just cause I’m so lucky now to have this community thing and get to be able to speak about something that I love, and write it off as a tax expense. That’s wonderful. So if I could get over more often, I probably would. But I go once a year for work, and then I try to come over once a year with my wife. So like next is Germany in January. She adores the band a-ha, and they’re doing a new tour, so we’re going over to see them in Germany in January.\nKN: Ok, this is your first time in Poland?\nBO: Yes!\nKN: How do you like it?\nBO: I mean the food is great. I had not thought about how good Polish food is! Like I love gnocchi, I love pierogis, I’m sure I’m mispronouncing all of that.\nKN: Very good, like Wrocław. [laughs]\nBO: Wrocław! I worked on that, so proud of that! But the food I was really surprised by. I hadn’t thought about how much I like Polish food before! And the scenery is\u0026hellip; this is what Americans imagine when they think of Europe. When they think of Europe, they think of an old, beautiful city, lots of trees, buildings that are a mix of a thousand years old and beautiful brand new architecture, you know, like the whole mix. Cobblestone streets that wind around, it’s just fantastic! I had heard from Aaron Bertrand and Grant Fritchey when they came over last year, they were like: “You have to come over here”. And I went: “Alright, let’s go see what it is”. And it’s beautiful, and we were talking a couple nights ago at the speaker dinner about how friendly everyone is here. The people were shocked by, they would just go and ask, I forgot which speaker it was that said he just ran into a policeman and he’s like: “Hey, can you tell me where whatever is?” The policeman starts bringing over other people, they navigate him to exactly where it is. You try this in America and they’re like: “Get out your phone, shut up, go off”. You’re on your own. [laughs] But yeah, it’s been beautiful, very nice.\nKN: Ok, Brent, thank you very much for this. It was a great pleasure. Thank you very much.\nBO: Oh, thank you! Thanks for organising the conference. You guys are doing a great job, it’s really well put together and the attendees are clearly having a great time, so nice job!\nDW: That’s also because of you, great speakers.\nKN: Yes, exactly! [laughs]\nBO: Oh, my pleasure! Thank you, guys.\nDW: Thank you.\nUseful links: Brent Ozar\u0026rsquo;s company: Brent Ozar Unlimited\nBrent Ozar\u0026rsquo;s Twitters: Private | Company\n","date":"2017-07-02T11:14:00Z","image":"/uploads/2017/07/KNDW_Interview_with_BrentOzar.jpg","permalink":"/2017/07/podcast-brent-ozar-interview/","title":"ASF 001: Brent Ozar interview"},{"content":"Almost one month ago has been ended (15-17 of May 2017) one of the biggest conferences in that part of Europe. Speech of course about SQLDay Conference in Wroclaw, Poland. The annual conference is taken every year about mid of may and is being organised by Data Community Poland.\nAs in a previous year, it has taken place in Centennial Hall (Hala Stulecia) and drew almost 800 attendees. It was a great event. We prepared 5 track within 61 regular sessions and 6 all day\u0026rsquo;s workshops during 3 days. Our main stars during the conference were:\nGreg Low [WWW | @greglow] Brent Ozar [WWW | @BrentO] Chris Webb [Blog | @Technitrain] They were leading whole day workshop on the first day of the conference (find the subjects at the end of this post) and a few regular sessions later on.\nIt is a tradition from some time that after the second day of the conference there is a party!\nIt was a fantastic opportunity to had a chat with many great speakers, have fun including dance, drink a bit, eat a lot (whichever you prefer and in whatever order) and so on.\nAre you still wondering to attend next year? Don\u0026rsquo;t! As always it\u0026rsquo;s tonnes of knowledge, many great people to meet, great fun and good value for money. Put your comment below this post or send me an email - I will let you know when we gonna kick off a registration for the next SQLDay 2018.\nSessions \u0026gt; Videos on Channel 9 \u0026gt; Materials (soon again) [pdf-embedder url=\u0026quot;/wp-content/uploads/2017/06/SQLDay2017_materials.pdf\u0026quot;]\nSQLDay 2017 in pictures [srizonfbalbum id=4]\nmore pictures\n","date":"2017-06-10T18:24:29Z","image":"/uploads/2017/06/SQLDay-logo_2017.png","permalink":"/2017/06/sqlday-2017/","title":"SQLDay 2017"},{"content":"Link to this (past) event: http://www.sqlsaturday.com/616/\nPictures [srizonfbalbum id=15]\n","date":"2017-05-20T19:43:10Z","image":"/uploads/2017/05/sqlsat-616-kiev.jpg","permalink":"/2017/05/sql-saturday-616-kiev-pictures/","title":"SQL Saturday #616 Kiev (pictures)"},{"content":"Link to this (past) event: http://www.sqlsaturday.com/589/\nPictures [srizonfbalbum id=16]\n","date":"2017-02-25T19:53:53Z","image":"/uploads/2017/02/sqlsat-589-pordenone.jpg","permalink":"/2017/02/sql-saturday-589-pordenone-pictures/","title":"SQL Saturday #589 Pordenone (pictures)"},{"content":"Visual Studio is turning twenty – Microsoft celebrating two decades of Visual Studio!\nOn this occasion, they announce that Visual Studio 2017 will be released on March 7, with a livestreamed two-day launch event at https://launch.visualstudio.com.\nRead more: Source\n","date":"2017-02-10T23:37:15Z","image":"/uploads/2017/02/ferdinand-stohr-149422.jpg","permalink":"/2017/02/visual-studio-2017-launch-event/","title":"Visual Studio 2017 Launch Event and 20th Anniversary"},{"content":"Link to this (past) event: http://www.sqlsaturday.com/567/\nPictures [srizonfbalbum id=2]\n","date":"2016-12-10T14:48:02Z","image":"/uploads/2016/12/SS567.png","permalink":"/2016/12/sqlsaturday-567-ljubljana/","title":"SQL Saturday #567 Ljubljana"},{"content":"Link to this (past) event: http://www.sqlsaturday.com/534/\nBelow you can find photos from the first SQL Saturday in Poland:\n[srizonfbalbum id=1]\n","date":"2016-10-01T23:44:11Z","image":"/uploads/2017/03/IMG_2264.jpg","permalink":"/2016/10/sqlsaturday-534-krakow/","title":"SQL Saturday #534 Kraków"},{"content":"[srizonfbalbum id=3]\n","date":"2016-05-18T23:55:30Z","image":"/uploads/2016/05/logo_2016-2.png","permalink":"/2016/05/sqlday-2016/","title":"SQLDay 2016"},{"content":"Hoho\u0026hellip;. może podsumowanie to za dużo powiedziane. Na pewno jednak chcieliśmy wrzucić w jednym miejscu kilka linków dotyczących minionej konferencji we Wrocławiu.\nAby jednak nawiązać choć trochę do tytułu posta - poniżej garść statystyk.\nSQLDay 2015 był jak do tej pory największą konferencją dla pasjonatów SQL Server\u0026rsquo;a w Polsce:\nPonad 400 uczestników, 5 speakerów zagranicznych i 16 krajowych, Ponad 4000 minut (66h) wiedzy w formie warsztatów (5) i sesji (30) w trzech równoległych ścieżkach, Zjedliśmy ok. 300kg posiłków podczas lunch\u0026rsquo;u, Wypiliśmy kilka kegów piwa, dziesiątki butelek wina oraz niezliczone litry kawy, herbaty i napojów, \u0026hellip; \u0026hellip; i z dumą możemy powiedzieć, że BYŁO WARTO! Po każdej sesji pytaliśmy o Waszą opinię. Mogliście zagłosować wybierając jedną z trzech kart: zieloną, żółtą, czerwoną. Rezultaty są następujące:\n84,3% zielonych (bardzo się podobało) 13% żółtych (średnio) 2,7% czerwonych (słabo) To bardzo dobry wynik, nie sądzicie? Oczywiście nie osiadamy na laurach i z roku na rok staramy się podnieść poziom i jakość konferencji.\nNo dobrze, wystarczy tych liczb - powspominajmy (audio)-wizualnie. Poniżej znajdziecie linki do wielu materiałów, które powstały w trakcie i po konferencji.\nfilm_reel Zapisy video sesji photo Foto-reportaż presentation Materiały Ponadto zachęcamy do obejrzenia materiału filmowego promującego konferencję SQLDay, który został nakręcony w 2015 roku we Wrocławiu:\n","date":"2015-08-29T22:13:13Z","image":"/uploads/2015/08/DSC4466.jpg","permalink":"/2015/08/sqlday-2015-summary/","title":"SQLDay 2015 summary [PL]"},{"content":"[srizonfbalbum id=6]\n","date":"2014-05-04T22:51:15Z","image":"/uploads/2017/12/logo_SQL-2014_kolor.png","permalink":"/2014/05/sqlday-2014/","title":"SQLDay 2014"}]