Magento Test Automation

---
layout: true
.footer[Magento Test Automation: My Journey .twitter[@fschmengler]]
---

.header[
 .testrun[
 .now[2011] > 2012 > 2013 > 2014 > 2015 > 2016 > 2017
 .
 ]
]

--
.width50[![I have no idea](img/noidea.jpg)]

---
background-image: url(img/learning.jpg)
class: center,fullwidth

---
.header[
 .testrun[
 .now[2011] > 2012 > 2013 > 2014 > 2015 > 2016 > 2017
 ....
 ]
]

- SimpleTest? PHPUnit?
- Ibuildings MageTest? EcomDev PHPUnit?

---

---
background-image:url(img/what.jpg)

.shadow.huge.white[
**foo_bar? 
 Foo_Bar? 
 fooBar? 
 foobar? 
 bar? 
 foo/bar?**
]
---

# TEST ALL THE THINGS!

```
    public function testObserver()
    {
        $this->assertEventObserverDefined(
            'adminhtml', 'core_block_abstract_prepare_layout_before',
            SGH_OrderPrintPopups_Model_Observer_Block::MODEL,
            'onCoreBlockAbstractPrepareLayoutBefore', 'orderprintpopups'
        );
    }
```

---

# TEST ALL THE THINGS!

```
    /**
     * @param Mage_Core_Block_Abstract $block
     * @return Varien_Event_Observer
     */
    protected function _mockEventObserver($block, $otherArgs = array())
    {
        $args = array_merge(array('block' => $block), $otherArgs);
        $observer = new Varien_Event_Observer();
        $observer->setEvent(new Varien_Event($args));
        return $observer;
    }
```

---

# TEST ALL THE THINGS!

```

$this->assertLayoutHandleLoaded(
        'orderprintpopups_warensendung_index',
        'module layout handle loaded'
    );
    $this->assertLayoutBlockRendered(
        SGH_OrderPrintPopups_Block_Popup_Warensendung::ALIAS,
        'popup block rendered'
    );

```

---

# TEST ALL THE THINGS!

```
 $route = 'orderprintpopups/warensendung/barcodestatic';
 $this->assertResponseBodyQueryRegex(
 '#sgh_warensendung',
 '#<img [^>]*src="[^"]+' . $route . '#'
 );

```

---

# TEST ALL THE THINGS!

```

/**
     * getter tests
     */
    test("getter", function() {
        	equal(this.block.getName(), "foo", "getName()");
        	equal(this.block.getContent(), "lorem ipsum falleri fallera", "getContent()");
        	equal(this.block.getPlaceholder(), "{foo}", "getPlaceholder()");
    });
```
---

# What lies behind the code

- Hours of debugging
- Hidden depenencies to global state
- Bugs of the testing frameworks

---

# A story of skipped tests

```
$this->markTestSkipped('fatal error???');
```
--
```
$this->markTestSkipped('conflict with fixture and existing db');
```
--
```
$this->markTestSkipped('"out of stock", although fixture seems ok'); //FIXME
```
--
```
$this->markTestSkipped(
    'The new comparator classes in PHPUnit 3.6 cannot handle recursive comparision with
    $maxDepth parameter YET.'
);
```
--
```
$this->markTestSkipped(
    'expectation + fixture changed. reason for missing prefix unknown '.
    'but only occurs in test environment!'
);
```
---

# A story of skipped tests

```
$this->markTestSkipped(
    'Das klappt schon wieder alles nicht mit Adminlogin+Dispatch'
);
```
--
```
$this->markTestSkipped('Untestbar in der Form. Mglw. ein Bug in EcomDev_PHPUnit der zu leerem
    orig_data führt. Leider funktioniert @runInSeparateProcess nicht mit Magento.')
```
---

.header[
 .testrun[
 .now[2011] > 2012 > 2013 > 2014 > 2015 > 2016 > 2017
 .............S[SSS].F[F]
 ]
]

##.black.shadow[Magento 1.6]
#.huge.black.shadow[Update]

---

.header[
 .testrun[
 .now[2011] > 2012 > 2013 > 2014 > 2015 > 2016 > 2017
 .............S[SSS].F[FF]
 ]
]

---
background-image: url(img/module-product-description-blocks.svg)

.header[
 .testrun[
 .now[2011] > 2012 > 2013 > 2014 > 2015 > 2016 > 2017
 .............S[SSS].F[FF].
 ]
]

---

.header[
 .testrun[
 2011 > .now[2012] > 2013 > 2014 > 2015 > 2016 > 2017
 .............S[SSS].F[FF]....S[S]
 ]
]

---

.header[
 .testrun[
 2011 > .now[2012] > 2013 > 2014 > 2015 > 2016 > 2017
 .............S[SSS].F[FF]....S[S].
 ]
]

---
class: fullwidth
background-image: url(img/thesis-unit-tests.jpg)

.header[
 .testrun[
 2011 > .now[2012] > 2013 > 2014 > 2015 > 2016 > 2017
 .............S[SSS].F[FF]....S[S]..
 ]
]

#.shadow[Diploma Thesis on Magento Development]

---

.header[
 .testrun[
 2011 > .now[2012] > 2013 > 2014 > 2015 > 2016 > 2017
 .............S[SSS].F[FF]....S[S]...
 ]
]

# Research & Experiments

- Built-in tests
- Magento TAF for Smoke Test Suite
- Abstraction of automated acceptance tests 
  .lineheight[![Fitnesse](img/fitnesse.png)] + ZiBreve

---

.header[
 .testrun[
 2011 > 2012 > .now[2013] > 2014 > 2015 > 2016 > 2017
 .............S[SSS].F[FF]....S[S]....
 ]
]

#.shadow[Selenium IDE > FitNesse]
---

.header[
 .testrun[
 2011 > 2012 > .now[2013] > 2014 > 2015 > 2016 > 2017
 .............S[SSS].F[FF]....S[S].....
 ]
]

# Agency project with Selenium

## Automated Acceptance Tests

- Selenium IDE .lineheight[![Selenium](img/selenium.png)]
- Tests for every feature 
- Partially written by QA

---
class: center

.header[
 .testrun[
 2011 > 2012 > .now[2013] > 2014 > 2015 > 2016 > 2017
 .............S[SSS].F[FF]....S[S]......S[S]
 ]
]

![Clock](img/clock.svg)
![Fragile](img/fragile.jpg)
---

.header[
 .testrun[
 2011 > 2012 > 2013 > .now[2014] > 2015 > 2016 > 2017
 .............S[SSS].F[FF]....S[S]......S[S].
 ]
]

# Agency project with EcomDev_PHPUnit

## Automated Unit/Integration T  ests

- Tests mandatory, coverage monitored
- Sophisticated setup
- Still typical problems

---

.header[
 .testrun[
 2011 > 2012 > 2013 > .now[2014] > 2015 > 2016 > 2017
 .............S[SSS].F[FF]....S[S]......S[S]..F[F]
 ]
]

# DEADLINE IS COMING

---

.header[
 .testrun[
 2011 > 2012 > 2013 > .now[2014] > 2015 > 2016 > 2017
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF]
 ]
]

# M1 Integration Tests: Typical Problems

- Tests altering global state (singletons, registry)

- Tests depending on existing data, others delete it and use their own
--

- Tests not well isolated, not running in DB transaction
--

- Complicated fixture setup

---

.header[
 .testrun[
 2011 > 2012 > 2013 > 2014 > .now[2015] > 2016 > 2017
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF].
 ]
]

- .green[**Decouple**] business logic from Magento
- Use plain PHPUnit

---

.header[
 .testrun[
 2011 > 2012 > 2013 > 2014 > .now[2015] > 2016 > 2017
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF]..
 ]
]

![yay](img/yay.png)

---

.header[
 .testrun[
 2011 > 2012 > 2013 > 2014 > .now[2015] > 2016 > 2017
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF]...
 ]
]

#.red[Bad news]

- we still need higher level tests

#.green[Good news]

- less of them

---
class: center

.header[
 .testrun[
 2011 > 2012 > 2013 > 2014 > .now[2015] > 2016 > 2017
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF]....
 ]
]

![test pyramid](img/agile-testing-pyramid.png)

---

.header[
 .testrun[
 2011 > 2012 > 2013 > 2014 > .now[2015] > 2016 > 2017
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF].....
 ]
]

# Experiments

- Acceptance and integration test framework .lineheight[![xtest](img/xtest.png)]
--

- Acceptance test framework ![Magium](img/magium.png)
  > "THE SELENIUM-BASED
TESTING FRAMEWORK FOR
PEOPLE WHO HATE TESTING"
--

- Test framework Codeception .lineheight[![Codeception](img/codeception.svg)]
---

.header[
 .testrun[
 2011 > 2012 > 2013 > 2014 > 2015 > .now[2016] > 2017
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF]......
 ]
]

##A new era:
#.huge.magentoorange[Magento 2]

---

.header[
 .testrun[
 2011 > 2012 > 2013 > 2014 > 2015 > .now[2016] > 2017
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF].......
 ]
]

# Magento 2 Extension

![IntegerNet_Solr](img/integernet-solr-logo.png)

- Reused refactored lib from Magento 1 extension
- Integration tests for Magento 2 specific code

---

.header[
 .testrun[
 2011 > 2012 > 2013 > 2014 > 2015 > .now[2016] > 2017
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF]........
 ]
]

# Magento 2 Projects

- Great integration test framework
--

- Easier to start with
--

- More reliable
---

.header[
 .testrun[
 2011 > 2012 > 2013 > 2014 > 2015 > .now[2016] > 2017
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF].........S[S]
 ]
]
# Magento 2 Projects

- Still frustrating sometimes
--

- Failures after minor version update
--

- Test runs not completely isolated

--
- .green[**Decoupled**] approach still works best

---

.header[
 .testrun[
 2011 > 2012 > 2013 > 2014 > 2015 > 2016 > .now[2017]
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF].........S[S].
 ]
]

# TDD Katas: Back to the roots

- Regular practice
- Small tasks, no real project
- **Goal:** Make TDD my *default habit*
- .white[`https://www.schmengler-se.de/katas`]

---

.header[
 .testrun[
 2011 > 2012 > 2013 > 2014 > 2015 > 2016 > .now[2017]
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF].........S[S]..
 ]
]

#How I would start today / Recommendations

0. For a .green[**quick win**], create smoke test suite with Codeception
--

1. Katas / small self contained projects to .green[**learn TDD**]
--

2. Start to build (small) parts of your code .green[**independent**] from Magento - "TDD" those parts
--

3. Try to add .green[**simple integration tests**] (in M1 with Xtest)
  - but .red[skip] if it's getting .red[complicated]

---

.header[
 .testrun[
 2011 > 2012 > 2013 > 2014 > 2015 > 2016 > .now[2017]
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF].........S[S]...
 ]
]

#Mistakes I won't make again

- Try to .red[sell tests] as separate part of project quote or estimation

--
- Try to teach "unit testing" and "testing Magento" .red[at once]

---

.header[
 .testrun[
 2011 > 2012 > 2013 > 2014 > 2015 > 2016 > .now[2017]
 .............S[SSS].F[FF]....S[S]......S[S]..F[FF].........S[S]...
 
 Time: 40 minutes
 
 .OK[OK (42 slides, 0 cat pictures)]
 ]
]
--

#.center.big[Thank you!]

##.center.big[Questions?]
---
background-image: url(img/blackmetalcat.jpg)
class: center
---
class: center
#Announcement:
#.huge[MageTestFest]

NOVEMBER 17TH 2017

`https://magetestfest.nl/` *by Yireo. *

---

# Links
.grey[
- `http://xtest-mage.com/`
- `https://magiumlib.com/`
- `http://codeception.com/`
]

# Image sources
.grey[
- Marvin: The Hitchhiker's Guide to the Galaxy (2005 movie)
- Angry German: https://www.youtube.com/watch?v=-_xUIDRxdmc
- Black metal cat: https://twitter.com/evilbmcats
- Agile testing pyramid: http://www.agilecoachjournal.com
]