# Scala algorithm: Remove duplicates from an unsorted List

Published

## Algorithm goal

Remove duplicates from an unsorted list; for example, [1,2,1,2,3] becomes [1,2,3] because the 1 can only be repeated once, the 2 can only be repeated once, and the 3 is only repeated once. Use cases here are especially in streaming and processing idempotent data, such as that in unreliable networks where packets or messages may be sent multiple times.

## Test cases in Scala

``````assert(removeDuplicatesUnsortedList(Nil) == Nil)
assert(removeDuplicatesUnsortedList(List(1)) == List(1))
assert(removeDuplicatesUnsortedList(List(1, 2)) == List(1, 2))
assert(removeDuplicatesUnsortedList(List(1, 2, 1)) == List(1, 2))
assert(removeDuplicatesUnsortedList(List(1, 2, 1, 2)) == List(1, 2))
assert(removeDuplicatesUnsortedList(List(1, 2, 1, 2, 3)) == List(1, 2, 3))
assert(
removeDuplicatesUnsortedList(List(1, 2, 1, 2, 3, 1)) == List(1, 2, 3)
)
``````

## Algorithm in Scala

17 lines of Scala (compatible versions 2.13 & 3.0), showing how concise Scala can be!

## Explanation

Using a scanLeft and scanRight, we can do what would typically be a while-loop, however we do it in a declarative way where we check if the element has been seen, and if not, then we emit it as a Some(); otherwise we emit a None to say no value has been emmitted.

This is very close to a State machine, and in fact can be extracted out as such. See: RemoveDuplicatesFromSortedListStateMachine, ParenthesesFoldingStateMachine. (this is Â© from www.scala-algorithms.com)

## Scala concepts & Hints

1. ### Collect

'collect' allows you to use Pattern Matching, to filter and map items.

``````assert("Hello World".collect {
case character if Character.isUpperCase(character) => character.toLower
} == "hw")
``````
2. ### Lazy List

The 'LazyList' type (previously known as 'Stream' in Scala) is used to describe a potentially infinite list that evaluates only when necessary ('lazily').

3. ### Option Type

The 'Option' type is used to describe a computation that either has a result or does not. In Scala, you can 'chain' Option processing, combine with lists and other data structures. For example, you can also turn a pattern-match into a function that return an Option, and vice-versa!

``````assert(Option(1).flatMap(x => Option(x + 2)) == Option(3))

assert(Option(1).flatMap(x => None) == None)
``````
4. ### Pattern Matching

Pattern matching in Scala lets you quickly identify what you are looking for in a data, and also extract it.

``````assert("Hello World".collect {
case character if Character.isUpperCase(character) => character.toLower
} == "hw")
``````
5. ### scanLeft and scanRight

Scala's `scan` functions enable you to do folds like foldLeft and foldRight, while collecting the intermediate results

``````assert(List(1, 2, 3, 4, 5).scanLeft(0)(_ + _) == List(0, 1, 3, 6, 10, 15))
``````
6. ### Stack Safety

Stack safety is present where a function cannot crash due to overflowing the limit of number of recursive calls.

This function will work for n = 5, but will not work for n = 2000 (crash with java.lang.StackOverflowError) - however there is a way to fix it :-)

In Scala Algorithms, we try to write the algorithms in a stack-safe way, where possible, so that when you use the algorithms, they will not crash on large inputs. However, stack-safe implementations are often more complex, and in some cases, overly complex, for the task at hand.

``````def sum(from: Int, until: Int): Int =
if (from == until) until else from + sum(from + 1, until)

def thisWillSucceed: Int = sum(1, 5)

def thisWillFail: Int = sum(1, 300)
``````

# Scala Algorithms: The most comprehensive library of algorithms in standard pure-functional Scala

## How our 100 algorithms look

1. A description/goal of the algorithm.
2. An explanation with both Scala and logical parts.
3. A proof or a derivation, where appropriate.
4. Links to Scala concepts used in this specific algorithm, also unit-tested.
5. An implementation in pure-functional immutable Scala, with efficiency in mind (for most algorithms, this is for paid subscribers only).
6. Unit tests, with a button to run them immediately in our in-browser IDE.

### Study our 100 Scala Algorithms: 6 fully free, 100 published & 0 upcoming

Fully unit-tested, with explanations and relevant concepts; new algorithms published about once a week.

### Explore the 22 most useful Scala concepts

To save you going through various tutorials, we cherry-picked the most useful Scala concepts in a consistent form.

## Subscribe to Scala Algorithms

Maximize your Scala with disciplined and consistently unit-tested solutions to 100+ algorithms.

Use it from improving your day-to-day data structures and Scala; all the way to interviewing.